Research Into Real-Time Video Monitoring and Remote-Control System of Escalators

In this Engineering article, your author explores means to improve safety.

by Yantai Luo

Keywords: video monitoring; detection; functional safety; remote control

Abstract:

Escalators are safety-related, heavy-duty special equipment. It is a challenge for escalator manufacturers to detect a dangerous state or the behavior of passengers on an escalator by using real-time video monitoring, which could help the administrators in a security-monitoring room control the escalator remotely at the emergent moment. This hasn’t been solved effectively since the escalator was invented 120 years ago. This paper presents some practical methods to overcome the difficulties, which include these two aspects:

1) Detect a dangerous state or behavior of passengers, such as accidental falls, or walking in the reverse direction, based on real-time video analysis. To improve the accuracy of detection, this article introduces the method of real-time video analysis which combines optical flow algorithm and image deep-learning algorithm.

2) Based on functional safety technology, use only one emergency-stop button in the security monitoring room to control any of the escalators on which the dangerous state of the equipment or behavior of passengers is detected remotely.

The actual run of the system in metro stations demonstrates it can detect dangerous equipment conditions or behavior of passengers accurately and stably, and administrators in security-monitoring rooms can stop the escalator in a safer and more convenient manner. The application result shows that the proposed system offers significant value for the lift industry.

Introduction

Escalators are widely used for vertical-transportation of people in subway, light rail and train stations, airports, wharves, shopping malls, etc.

In China, with the rapid development of urban subways, the use of escalators for subways has increased correspondingly. Take Shanghai Metro, the largest urban rail transit network in China, as an example: At the end of 2018, the total length of the network of Shanghai urban rail transit reached 705 km, ranking first in the world, and recorded 2.038 billion passenger trips in the year, ranking third.[1] The Shanghai Metro has more than 4,000 escalators, which carry more than 30 million passengers a day and bear more than 50% of Shanghai’s public-transportation capacity.^[2]

Due to the inevitable contact of moving parts with passengers, injuries on escalators account for more than 60% of the total number of rail-transit passenger injuries: up to 80 per month.^[3] According to statistics, falls account for about 75% of all types of escalator passenger injuries.^[4] Therefore, detecting a fall on the escalator as it happens, and stopping the escalator remotely and safely by the staff in the control room, can help prevent major personal injury. A moving escalator can cause continuous rolling of fallen passengers, an urgent problem to be solved in the daily operation management of the escalator.

Methods

Real-Time Video Analysis

Existing escalator detection systems can only detect the operation of traditional machinery, so the monitoring of passenger behavior mainly depends on video surveillance with manual real-time viewing. Because of the wide distribution of escalators, the large number to be monitored and the lack sufficient staff to implement real-time monitoring, video surveillance systems usually can only play the role of passive tracing after the accident.

With the development of AI image recognition technology, the real-time video-analysis-related research of escalator passenger behavior is booming. For example, based on the Openpose^[5] key point detection algorithm to obtain the human body structure, and the use of the support vector machine (SVM) classifier to realize the detection and alarm of the pedestrian fall phenomenon in the picture,[6] the defect of the algorithm is that it is necessary to detect the key point of the human body, which is easily blocked by the rear passenger or object, and when classifying the human body posture by the SVM, it is difficult to complete the accurate discrimination. The algorithm is based on the human skeleton sequence of the escalator abnormal behavior recognition algorithm. The algorithm firstly detects the passenger’s face by using the SVM, which combines the features of the deformable component model and tracks the passenger’s motion in the escalator with an improved kernel correlation filter. Then, the human skeleton sequence of the passenger is extracted by using the convolution neural network, and the abnormal behavior sequence is detected from the passenger skeleton sequence by template-matching. Finally, the algorithm uses the dynamic time-regularization to identify the abnormal behavior sequence based on the five-nearest-neighbor method. The optical flow-tracking algorithm and the human object are tracked to define the related abnormal behavior, and the abnormal behavior detection is carried out.^[7] This algorithm only determines the fall through the optical flow tracking, and the misjudgment rate is high.

To achieve a high-accuracy, low-misjudgment rate, and a low-cost real-time video method to detect passenger falls on the escalator, this paper proposes a method to first extract the direction of passenger motion based on the dense optical flow method, then group the information eigenvalues of each sub-block motion direction, speed, acceleration and so on in the block that accords with the pixel points in the direction of motion, and then use the SVM classifier to classify the eigenvalues of the matrix in the continuous 1 to 2 s to determine whether the pedestrian in the picture is normal, reverse-walking or falling.

The Calculation Process Is As Follows:

1. Obtain the first frame image detected by optical flow method
2. Get the second frame image as the current frame image detected by optical flow method
3. The motion direction and displacement of each pixel in the image are calculated by dense optical flow method.
4. Calculate the area of the moving block and the speed of the moving block with the escalator in the third step optical flow diagram displacement distance, mean velocity and variance.
5. Cache the calculated results of step 4 as the feature vector of this frame.
6. Repeat steps 2, 3, 4, 5 using the current frame as the first frame for detection.
7. To judge the number of frames with the current pre-frame N frame motion block area > MIN_S, when the number of frames > K, the preliminary judgment is suspected to be a fall or reverse-walking.
8. The feature matrix composed of N frames was input into the SVM classifier for classification to determine whether it was fall, reverse-walking or false alarm.

Feature Generation and Training Specific Algorithms:

Single Frame Feature Generation:

• According to the optical flow calculation, the displacement maps of the x-direction and y-direction of the image are obtained respectively:

The downward displacement should be greater than the left and right displacement according to the retrograde and fall movement characteristics of the escalator. Obtain the motion block mask image from the above displacement map:

• Area extraction of moving blocks:

Open operation of mask image; calculate mask m_ij = 1   calculate the area of the connected domain as S.

Calculation of Average Vertical Displacement Velocity

Calculate the average velocity of the moving block according to the motion block mask map mask, displacement map in the y direction.

• Velocity variance calculation, based on the average velocity calculated in step 3 to obtain velocity variance:

Combination of 25-Frame Single-frame Features:

The feature vector of 25 consecutive frames is 25*3=75 eigenvalues.

• The elements in the three groups of vectors in F are arranged in ascending order respectively to get .
• Collect feature vectors  in each case for SVM classifier training.
• Save the classifier’s training results for subsequent classification use.

Stop Escalator Remotely

Since it was invented more than 120 years ago, the operation mode of escalators is all local operation, and the practice of attempting remote operation by subway companies has not been successful. When there is an emergency at the scene (passenger falls on the escalator, congestion at the escalator exit, etc.), the situation can rely only on scene personnel finding the on-site emergency operation. To ensure the safety of operation, some subway operation companies send more personnel during peak periods and emergency intervention.

• In the Moscow Subway (Figure 1), at the side of the escalators, a guard box and an operator on duty are set up. The operator observes the escalators and pushes the emergency stop switch immediately in case of an emergency.
• Domestic portion MTR Corporation (China, Figure 2): In the peak hour of danger, temporary staff is on duty; if an emergency occurs, the on-duty staff initiates an Emergency Stop to intervene.

• Domestic portion MTR Corporation (China, Figure 2): In the peak hour of danger, temporary staff is on duty; if an emergency occurs, the on-duty staff initiates an Emergency Stop to intervene.

If we can realize remote control of escalators in the security staff monitoring room, especially with intelligent awareness and safe, remote control of emergency events, this will greatly accelerate emergency responses and reduce personal injury. But, as specialty equipment, remote control of escalators must be strictly compliant with security standards and regulations. Otherwise, if there is a software defect, component failure or electromagnetic compatibility problem, a remote response will fail to implement an emergency stop correctly, or cause an incorrect emergency stop, which will lead to a more dangerous situation. The emergency-stop device shall be an electrical safety device compliant with EN115-1:2008+A1:2010 5.12.1.2, i.e.: it may consist of A) either one or more safety switches, or B) safety circuit (fail-safe circuit), or C) programmable electronic safety-related system (PESSRAE).

Currently, the world’s response strategy is like type A, with a physical emergency-stop button for each escalator that conforms to the electrical safety device (EN115-1:2008+A1:2010 5.12.1.2). Operators manually press the corresponding physical emergency-stop button (Figure 3) to stop the escalator remotely in the event of a passenger fall.

This approach has the following disadvantages:

• It’s not easy to find accurate buttons in an emergency.
• When the operation error causes further injury, the operator should bear the corresponding safety responsibility, so that the operator is not willing to use it, or wouldn’t dare use it.^[8]

This paper proposes a safe and convenient way to stop escalators remotely (i.e., mode C). Operators press only one actual stop button to cut off the safety circuit and stop the escalator — on which the dangerous event occurs — through the control of video analysis server and remote control components (which pass PESSRAE safety certification and obtain type test certificate; Figure 4).

The operating procedures are as follows:

1. Video-analysis server and remote-control components preserve the escalator information when detecting that a passenger is in a dangerous condition on an escalator;
2. Monitor in the security-monitoring room displays corresponding real-time video according to the information preserved in the video-analysis server and remote-control components;
3. The operator manually confirms that the escalator information is consistent and presses the actual stop button if there are dangerous conditions on the escalator in the video;
4. Video-analysis server and remote-control device components cut off the corresponding safety circuit of the escalator to stop the escalator.

Results

The real-time video-monitoring and remote-control system, which includes a total of eight cameras corresponding to 19 escalators, runs from August 2018 to October 2019 In Shanghai Metro Hanzhong Road Station.

1. Successfully capture:
   1. 32 passenger falls (all passenger falls were successfully captured)
   2. 738 passenger reverse-walking events
2. Realize safety remote control:
   1. The further personal injury of the passenger is effectively prevented, and safety is greatly improved.

Take a case that happened at 3:20 p.m. on August 17, 2018, as an example on an upward-running escalator in the Shanghai Metro Hanzhong Road Station. In this case, an older woman falls down because of failure to grasp the handrail. The operator in the security monitoring room is informed by the monitor and presses the actual emergency button to stop the escalator immediately, as shown in Figures 5 and 6. The time is 14 s from the time the fall happened until the escalator stopped; the time is 26 s from the time the fall happened until the counsel in the station arrived at the escalator.

The results show that the system has high accuracy and low misjudgment rate, and the remote-stopping escalator is safe and convenient. It offers wide application prospects in the lift industry all over the world.

References

[1] Liu Chunjie. “Research and Practice of Shanghai Smart Subway [J].” Urban Rail Transit Research, 2019 (6): 1-6
[2] Liu Chunjie. “Innovation and Practice of Intelligent Control of Shanghai Rail Transit Escalator [J].” China Municipal Engineering, 2019 (06): 1-3+100.
[3] Long, Sijin,Lu, Jian,Xing, Yingying, et al. “An analysis of escalator-related injuries in metro stations in China, 2013-2015[J].” Accident Analysis and Prevention, 2019, 122 (Jan.): 332-341.
[4] Filippone, J., Feldman, J.D., Schloss, R.D. and Cooper, D.A.(2002). “Elevator and Escalator Accident Reconstruction and Litigation,” Lawyers & Judges Publishing Company, Inc.
[5] M. Andriluka, S. Roth, and B. Schiele. “Monocular 3D pose estimation and tracking by detection.” CVPR, 2010.
[6] Chen Dong. “Software development of pedestrian safety monitoring system for electric escalators based on video analysis[D].” Zhejiang University,2019.
[7] Tian Lianfang, Wu Qichao, du Qiliang, Huang Liguang, Li Miao, Zhang Daming. “Identification of the abnormal behavior of walking elevator passengers based on human skeleton sequence[J].” Journal of South China University of Technology (Natural Science), 2019, 47 (04): 10-19.
[8] Yang Guanbao. “Escalator energy saving and intelligent monitoring system based on panoramic vision[D].” Zhejiang University of Technology, 2011.
[9] He Cheng. “AI Image Recognition and Functional Safety-Based Escalator Supervisory and Control System and Related Safety Regulations[J].” China Elevator, 2019, (15): 6-8.