Perception through Image Processing Applications for Situation Awareness

This paper introduces a concept for the perception level of situation awareness through image processing applications. The perception of the information from the environment is lacking to achieve highly autonomous vehicles. This approach mainly focuses on collecting information from the environment using camera sensors. The frames from the camera are processed using multiple image processing algorithms, outputs converted into CAN message, and sent to the expert system. CE-Box custom hardware that consists of a Raspberry Pi 3b model counted with PiCAN 2 used to implement and evaluate this approach. The goal of this paper is to provide a conceptual method to make decisions based on the extracted information from the environment using image processing algorithms. Keywords— Situation Awareness, Perception, CE-Box, Raspberry Pi, PiCAN 2

I. MOTIVATION As we move up the ladder of autonomous driving that is, from Level 0 to Level 5 the complexity of the software increases. Endsley M.R. highlighted in her paper [2], automotive companies have a software failure rate of 82 percent. Most of this failure lies due to the lack of understanding of surrounding situations of the vehicle. Collecting the information to identify the driving situation will further intensify the software's complexity. This has been the motivation to develop the concept of the classification of the driving situations. The classification of the situation might help to decrease software complexity. The Situation Awareness [1] is defined as how the environmental elements or events are perceived, comprehended the meaning and future actions are projected with respect to space and time.
There are three levels in situation awareness [3].
(i) Perception: Observing and attention to the information such as dynamics, features, and status which are important and relevant to the situation [3].
(ii) Comprehension: All the information are processed. This also involves the integration of the situation's complete picture from different elements which results in the understanding the importance situation [3].
(iii) Projection: Reacting according to the result of comprehension in a timely manner. This also predicts future actions on the source of the situation comprehension [3].
Situations are the features extracted from the external environments and from the vehicle itself. These features need to be perceived to understand the situation and make decisions accordingly. The rules for the expert system can be created based on traffic rules and regulations. This paper proposes the conceptual perception model using various image processing applications and machine learning approaches.
Every image processing algorithms are implemented on a separate Raspberry Pi mounted with PiCAN 2 of the CE-Box. The Raspberry Pis in the CE-Box are connected with the Adhoc network. The master Raspberry Pi access all the slaves using the SSH protocol and feeds the video input to each slave wirelessly using the SCP protocol. This concept was evaluated in a prototypical hardware unit CE-Box and is based on vision-based approaches.

A. Situational Awareness
Endsley, M.R. [4], situation awareness in the system operation is one of the major concern in which decisions are made based on an eloquent view of the different state is proposed. Here initially, several individual and environmental factors are explored and their relation between situation awareness. The situation awareness deals with the complexity of system, features and operator automation. The model of the situation awareness introduces the classification of errors and design implications are generated to enhance the operator situation awareness.
Rashaad E.T. et al. [5], presented a paper on the situation awareness levels. The first level perceives the important factors in the environment, the second level tries to recognize the meaning of the perceived factors, and finally, the third level will predict the near future with the perceived situation.

B. Expert System
Niehaus A. et al. [6] assumed that vehicle consists of sensors for the detection of traffic, signs, geometry of the road, and also actuators and control logic for controlling the brakes, steering wheel's angle and throttle. The expert system is given with the situation of the traffic, signs and driver's strategy such that commands are issued to the controllers. The system comprises of a rule base that provides the required knowledge for driving. The reasoning was performed by backward chaining inference, and the reasoning process is optimized by the compiler. The vehicle is controlled either by the expert system or preset strategy.
Charles L.F. [7] presented about Rete Match Algorithm in this article. This algorithm is very efficient in comparing the massive collection of patterns and objects as not iterating over the sets. The iteration over the patterns does not occur in this program as it contains a network which works on treestructure sorting or pattern index and efficiently matches all the objects with the pattern. The algorithm is used as the interpreters in the production system and used for the systems which consist of hundreds or thousands of objects and patterns.

C. Client-Server Communication
The Client-server communication accomplished using three different methods as presented by Zhao C.W. et al. [10]. In the first method, short-range communication accomplished using the ZigBee protocol. In the second method, uses Raspberry Pi's wifi-enabled by the samba software. In the third method, the socket connection used to transmit messages between host and client. The node-tonode communication between two Raspberry Pis performed based on the socket program.
In this study, a conceptual perception model for situation awareness is suggested with Image processing applications. The image processing algorithms utilize computer vision techniques and machine learning approaches. Both the designs are implemented and evaluated using the racks of hardware unit "CE-Box" of TU Chemnitz. The situation awareness deals with the complexity of system, features and operator automation. Several individual and environmental factors are collected in the perception level. From the perception level, collected elements and converted into facts which are used as the input for the comprehension level. The comprehension level will consist of a rule-based system which will match the element with the set of rules. The comprehension level will also predict the near future action. Finally, action commands are used to control the operation of the vehicle in the projection level [8].
The main steps of the conceptualization are [8]: 1. Getting the sensory data from the ECU through PiCAN 2. Getting the Image Processed Data (IPD) from the Raspberry Pi camera which is controlled by the Raspberry Pi Camera Controller (RCC) 3. Assembler will generate fact list from the raw data 4. Facts are matched with the predefined rules in the Expert System Shell (EXS) 5. Expert system will update the fact and give the command request to the assembler 6. Assembler will send the command as CAN message to the ECU 7. ECU executes the command. All these steps will run continuously in a cyclic manner

A. Sensory Information
The automotive vehicles includes different sensor which radar, rain sensor, proximity sensor, Speed sensor etc. The collected information from sensors through ECU are perceived and generated as facts. The generated facts are fed into Expert System. The Expert System will match the perceived facts with pre-defined rules and give command to the ECU through Assembler.

B. Image Processing Application
Along with the presence of different sensors in the automotive vehicle, a camera is also an important visual sensor. A camera provides video input that is used for detecting traffic sign, traffic light, lane, pedestrian etc. The primary dilemma we face here is that a level 5 car should simultaneously process all the algorithms so that it can utilize their outputs for actuation. The machine learning algorithm are implemented to perceive various situation of the environment. The perceived information from the camera are generated into facts and fed into the Expert System.

IV. IMAGE PROCESSING APPLICATIONS
The Raspberry Pi's in the CE-Box are connected based on Adhoc Network, shown in the figure 3. Master Pi can access all the slave Pis using the SSH protocol. Also, the master Pi uses the SCP protocol to broadcast video feed wirelessly onto all slave Pis simultaneously. The image processing algorithms in the slave Pis are trigger at the same instance by the master, and then the outputs of these algorithms are placed on to CAN bus.
The image processing algorithms can be further optimized by separating the pre-processing and improve performance. The number of slave Pis with the image processing algorithms can be added to the master. The same Adhoc network used to connect multiple CE-Box with multiple racks, such that more image processing algorithms can be adapted to comprehend the situation precisely. The hardware units like Raspberry Pi 4 or Jetson Nano, has more processing power than the used Raspberry Pi 3b model and increases the frame per second of the algorithms without changing the core architecture of the CE-Box.
The target of this system was to develop image processing applications for the recognition of traffic objects in given input videos and provide sensory sensor data. The videos are given as the input for the system architecture layer, whereas the processed data will be provided as output from the application layer. Several algorithms are performed simultaneously along with Graphical User Interface (GUI) to generate the sensory information. Once the algorithm detects the traffic object or sensory data, the separate command is created and a CAN message is provided to the ECU. This can be achieved by mounting PiCAN 2 on the Raspberry Pi.
The system architecture layer is presented in Figure 2. The implementation has been carried out in three stages ( Figure 3). In the first stage, the image processing / machine learning algorithm are implemented to detect and recognize traffic light [9], traffic speed sign, autobahn sign, pedestrian, and lane. In the second stage, GUI has been developed to depict the car's dashboard in order to provide synthetic sensory data. Finally, the perceived information from the above two stages has been converted into facts and fed into the Expert System. The Expert System will take action based on the pre-defined rules and give commands to the ECU. The ECU will act upon the command for example, like reducing speed, which can be seen in the GUI developed.

V. CURRENT STATE & FUTURE ASPECTS
The current system is at a prototypical state. Through this state, a prototypical cycle of Situation Awareness has been achieved, although in a very basic level. Due to the Raspberry pi 3b+ model is used as the target hardware, high level of performance was not achieved, especially when parallel image processing applications were performed. Optimizing these algorithms would be the next agenda to improve the performances. For the sake of evaluation, different machine learning and deep learning techniques are also being worked into. On the other hand, although several rules for expert system are created in order to test these perception results, these are not optimal. Rules based on the references from NHTSA and German traffic rules and regulations are being created currently. In the future, once the algorithms are optimized and diverse rules are generated for different scenarios, a significant improvement in Situation Awareness could be expected.