The Udacity SDC Engineer Nanodegree Term 1 begins with using basic computer vision algorithms to detect lane lines from a video and display them on screen.
Problem Statement
In this project, you will use the tools you learned about in the lesson to identify lane lines on the road. Once you have a result that looks roughly like “raw-lines-example.mp4”, you’ll need to get creative and try to average and/or extrapolate the line segments you’ve detected to map out the full extent of the lane lines. You can see an example of the result you’re going for in the video “P1_example.mp4”. Ultimately, you would like to draw just one line for the left side of the lane, and one for the right.
The tools you have are color selection, region of interest selection, grayscaling, Gaussian smoothing, Canny Edge Detection and Hough Tranform line detection. You are also free to explore and try other techniques that were not presented in the lesson. Your goal is piece together a pipeline to detect the line segments in the image, then average/extrapolate them and draw them onto the image for display (as below).
Solution and Write up
I will copy and paste sections from the iPython notebook that I made for this project. The Full project can be found on my github.
1. Describe your pipeline.
My lane line detection pipeline consists of the following steps:
- Convert each video frame to grayscale.
- Threshold the grayscale image.
- Apply region of interest to where lane lines are expected to be.
- Perform Canny edge detected (dilated here for easier viewing).
5. Perform the Hough transform (here with lines drawn).
6. Extend hough transform lines and bin them by positive and negative slopes:
Extending is done by taking the two tuples (x1,y1), (x2,y2) output from houghLinesP(), determining the slope for each line, and then determining a leftmost and rightmost point across the image to draw a line between.
7. Determine if both lane lines are detected, if one is not, draw one that is offset from the other lane line by a nominal amount so it is a function of the detected line.
8. If both lane lines are detected, perform least squares fit on positive and negative sloped lines:
9. Draw these least squares fit lines on an empty array and perform region of interest trimming to get rid of line segments extending beyond the horizon.
10. Overlay these lane lines on the original image and display them:
2. Identify potential shortcomings with your current pipeline
Potential shortcomings arise from situations where lane lines do not have high contrast with their background, when lines are occluded, or they are shadowed. For these situations there is no recovery behavior and vision can only be of so much assistance.
Another shortcoming is that if lane lines become too misaligned from the slopes expected they will not be registered as lane lines due to the binning process I, have which sorts things by their determined slope.
3. Suggest possible improvements to your pipeline
One large improvement would be to low pass or kalman filter the positions and slopes of the lane lines so that they are not memoryless like they currently are. This would dampen (add inertia) to the system and cause the lines to jitter much less.
Another potential improvement could be to filter the color image according to an RGB mask, as opposed to my simple grayscale threshold.
A further improvement would be to monitor the residual error terms or the covariance of the least squares line fitting to determine how much certainty there is in the fit. This could be used to monitor the state of the detection algorithm, and if the covariance becomes large some kind of recovery behavior within the code could be triggered.