Artificial intelligence-aided intelligent obstacle and trespasser detection based on locomotive-mounted forward-facing camera data

Huixiong Qin, Asim Zaman, Xiang Liu

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


Most fatalities in the railroad industry are trespasser deaths. Additional obstacles such as fallen trees, construction materials, and other debris also pose a danger to railroad operations. Understanding where, when, and how these events occur can help railroads develop trespasser mitigation strategies and improve rail safety. However, obtaining information from forward-facing footage requires immense manual effort. We developed a forward-facing trespassing and obstacle detection system that utilizes artificial intelligence to recognize and understand unsafe events and logs them for later review and analysis. The system dynamically identifies regions of interest, trespassers, and obstacles with a customized detection methodology which uses a semantic segmentation model called DeepLab and an object detection model called YOLOv5. These models were trained by a dataset containing over 10,000 images from various sources, including open-source datasets and videos provided by industry collaborators. The novelties of this research are threefold: the development of an algorithm capable of detecting the railroad’s restricted area around the track area, optimizing the trade-off between accuracy and latency to achieve real-time performance, and proposing a universal obstacle detection algorithm. The final system was able to analyze 400-pixel x 400-pixel videos at a rate of 14.5 frames per second (FPS) in an edge-computing device which identified trespassers and obstacles with more than 92% accuracy. The system can adaptively detect both obstacles and trespassers in the train’s path, as well as the railroad property area surrounding the tracks. This research offers the railroad industry tools which can provide precise trespasser and obstacle information to improve railroad safety and operations.

Original languageEnglish (US)
Pages (from-to)1230-1241
Number of pages12
JournalProceedings of the Institution of Mechanical Engineers, Part F: Journal of Rail and Rapid Transit
Issue number9
StatePublished - Oct 2023
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Mechanical Engineering


  • Railway
  • artificial intelligence
  • forward-facing footage
  • obstacle detection
  • trespasser detection


Dive into the research topics of 'Artificial intelligence-aided intelligent obstacle and trespasser detection based on locomotive-mounted forward-facing camera data'. Together they form a unique fingerprint.

Cite this