Train Detection Using OpenCV

It has been a while……a long while……since I’ve done an update on TrainAlert.  This is not due to lack of activity.  In fact, my lack of updating here is because of the effort that has been going into TrainAlert.  (For the unitiated:  TrainAlert is a solution, currently under development, to identify when a train is approaching the town where I live so subscribers can “get on the right side of the tracks” before the inevitable traffic disruption occurs.)
 
The primary challenges in getting this solution built have been 1/ finding a robust technology set upon which to base the solution and 2/ unlearning my 30 year old programming knowledge and applying modern software techniques.  The first challenge was addressed thanks to a friend who is interested in Computer Vision technologies.  He pointed me to OpenCV, an open source library of image analysis software routines.  The second challenge was addressed with time, Google and helpful people on the internet.  This blog post is not about the upcoming release of the solution per se.  Instead it’s about the stunning capabilities available in OpenCV.  OpenCV is yet another fantastic example of what the open source model can deliver.  So if you’re interested in kind-of-geeky Computer Vision topics……read on.
 
OpenCV can perform a broad set of tasks on digital images.  The images can be still pictures (i.e. photos) or frames from a video camera.  The train detection portion of TrainAlert is built using OpenCV routines, tied together by custom code written in Python to handle a variety of logic and analytic tasks.  

There are three primary OpenCV capabilities that are used to detect passing trains.  
 
Background Subtraction
The first step in identifying a passing train is finding items that might be a train.  Background Subtraction is the OpenCV process/method of identifying new objects in an image…..objects that aren’t normally there.  OpenCV’s background subtraction capability watches video frames over time (it keeps a history of the previous 5 video frames) and then identifies items in the current frame that are new.  It can’t identify what those items are.  They could be a variety of things.  But it can at least identify that something new is now in the frame.  When something new is found in a frame OpenCV then “subtracts out” the background portions of the image.  What is left is an image that contains only foreground objects (i.e. those items that are new). 
 
Contour Identification
Once it is known new objects are in the frame, they must be identified and characterized in some way.  OpenCV can provide size and location information about objects in images.  OpenCV calls these descriptors “contours”.  For example, OpenCV can draw a rectangle around each contour found in an image, and give back the details of those rectangles; most notably the coordinates of the rectangle’s four corners.  (OpenCV will return a list of rectangle coordinates for every contour in an image.)   
 
In the case of finding a train, a way to prioritize which contour might be most interesting is by its size.  Obviously, trains are pretty large objects.  So after knowing there are new objects in the frame, along with the sizes and locations of the foreground items, the TrainAlert program picks the largest frame contour and considers this the most likely "train suspect".  However the found contour is still just a suspect.  This new “large object” could be grass blowing in the wind.  Or vehicles that are visible in the distance.  Or clouds that are making dramatic movements.  Additional analysis needs to be done to determine if there really is a train in the frame or whether this large contour is some non-train anomaly.
 
​Object Tracking
One way to discern a train vs non-train contour is to see if and how it moves.  A train traveling down a track moves in a pretty predictable fashion.  The “big object” needs to be tracked to see if it is moving and, if so, by how much.  This is where OpenCV's Object Tracking capability is useful.  In order to track an object the TrainAlert program gives the OpenCV Object Tracker the coordinates of the large object that needs to be tracked.  OpenCV looks at the color profile found in that contour, and then looks for that same color profile in subsequent video frames coming off the camera.   When OpenCV sees that object, it delivers back to TrainAlert the coordinates for where that object is in that most-recently read video frame.  When this process is completed for a sampling of video frames, a location history will have been collected for where in the frame that object is over multiple video frames. 
 
At this point OpenCV has done all that it can do.  Now it's time for TrainAlert to decide "Is this a train or isn't it a train?"  In a simple and ideal world one looks at total distance traveled in the frame, compares the starting and ending positions, and concludes the train's direction of travel.  Unfortunately, the world is neither simple nor ideal.
 
It's not very easy to decide if there actually is a train or not.  The train is not always identified in the same portion of the frame.  And some non-train objects don't move like a train moves.  Wind blowing tree branches might result in movement data, but the data will likely be lots of "back-and-forth" and will not have consistent progression in the same direction over time.  Car lights in the distance definitely move in a consistent direction, but their movement is not as dramatic as train movement because they are farther away in the frame.  There are other non-train scenarios as well.
 
Here are two examples of object tracking.  The first shows how OpenCV tracked a real train.  The second video shows how OpenCV found "something new" in the frame, tracked it, but it turned to not be a train.

The trick in building this solution has been figuring out an algorithm that will look at the collected horizontal position values and decide whether the object in question is behaving as a train would.  Unfortunately it's not quite as simple as seeing if every data point shows progression in the same direction. The precision of Object Tracking is not perfect.  Even with actual trains Object Tracking will sometimes report either no movement or "backward" movement for one or two frames.   The tracked "bush" in the second video is easy to disregard, as total horizontal movement is in a very narrow window.  But other items that aren't trains don't behave in this way.  

I’ll spare the reader the gory details, but the bottom line is a variety of train and non-train object tracking data was put into a spreadsheet and patterns were identified.  Those patterns were identifiable with some simple calculations and, as a result, a method for determining train vs non-train scenarios has been developed.  Although these methods, too, are not perfect, current testing is showing an accuracy rate of greater than 90% for correctly identifying passing trains.
 
The train detection portion of TrainAlert is in final testing mode.  Without the OpenCV technology (and the help of others) I don’t believe this portion of the solution would be in the state it is today.  OpenCV is an astonishing technology that makes me very enthused about the types of solutions that are possible in the future.  It will be exciting to see what sort of Computer Vision applications become available.
 
So what is next for TrainAlert?  This article has been only about the train detection portion.  But there also needs to be a “subscriber notification” portion of TrainAlert – the infrastructure to deliver alerts to users.  Another generous friend is helping me on that front.  As that gets solidified more will be written here to describe how that works.  For now we’re marching ahead to finalize the first release – and help the citizens of Northern CO deal with the disruptive train traffic that plagues the area.
 
Jim