Lane detection system for day vision using altera DE2

The active safety systems used in automotive field are largely exploiting lane detection technique for warning the vehicle drivers to correct any unintended road departure and to reach fully autonomous vehicles. Due to its ability, to be programmed, to perform complex mathematical functions and its characterization of high speed processing, Field Programmable Gate Array (FPGA) could cope with the requirement of lane detection implementation and application. In the present work, lane detection is implemented using FPGA for day vision. This necessitates utilization of image processing techniques like filtering, edge detection and thresholding. The lane detection is performed by firstly capturing the image from a video camera and converted to gray scale. Then, a noise filtering process for gray image is performed using Gaussian and average filter. Methods from first and second order edge detection techniques have been selected for the purpose of lane edge detection. The effect of manually changing the threshold level on image enhancement has been examined. The results showed that raising threshold level would better enhance the image. The type of FPGA device used in the present work is Altera DE2. Firstly, the version DE2 Cyclone II start with (11xxxxxx-xxxx) together with Genx camera has been used. This camera supports both formats NTSC and PAL, while the above version of FPGA backups only NTSC format. The software of lane detection is designed and coded using Verilog language.


Introduction
Presently, road accidents are at the top leading causes of human being death, injury and disability.The loss of economic cost is also mainly attributed to road traffic accidents.It has been reported that the main reason behind most accidents are due to driver behavior.Falling asleep at the vehicle wheel, inattention and distraction (looking but cannot see) are the main identified causes of accidents [1].To avoid disorders of driver behavior, the vehicle motion is either controlled automatically or supplied by such means which can alarm the driver for any unexpected or potential collision.Therefore, different technologies and systems have been developed to reduce the road accidents such as road monitoring system, Global Positioning System (GPS), collision avoidance system, wireless communication, in a step avoid road accidents [2].
The lane detection technique can locate the lane boundaries of road or path geometry without a prior knowledge.The edge-based techniques are able to convert the detected edges into significant structures; even so they can find a suitable model for these edges.Many edgebased algorithms employ straight lines to formulate the lane boundaries, while others utilize more complicated models like parabola, B-Splines, hyperbola, etc. [3].Vision-based lane detection and departure warning system has been investigated over two decades.During this period, many different problems related to lane detection and departure warning have been addressed [3,4].Deep learning based lane boundary detection method that can accurately detect lane boundaries under complex weather conditions and traffic scenarios in real time [5].In this work, we present the implementation of lane boundary detection on hardware based on parallel processing to overcome time consuming.

Edge Detection Process
In the present work, the considered edge detection process comprises three stages; Noise filtering, edge detection techniques and thresholding.In what follows, a detail explanation will be presented [6].

Noise Filtering
Digital images are exposed to different types of noise.Noise occurs due to errors in the acquisition process of the image and its adverse effect is that the image pixel values do not represent the true intensities of the real scene.Various sources may cause noise in images.Noise may come from high temperature or transmission such as electronic circuit noise.Another source of noise is referred as quantization error which results from quantizing the pixels of sensed image into discrete levels.The noise can be introduced through brightening shadows or through color-balance processing [7,8].Many types of filters are considered in the image processing literature for removing the noise from an image; such as median filter, mean filter, high pass filter, average filter, Gaussian filter, inverse filter and Wiener filtering.In the present work, two famous filters have been only employed for real time edge detection; namely average and Gaussian filters [9].

Averaging Operator
Noise reduction is the main advantageous effect of averaging.On the other hand, the averaging could cause blurring which results in reducing the details in an image.Moreover, it is a low-pass filter; as it permits retention of low spatial frequencies and suppression of high frequency components.Larger template, say 3×3 or 5×5, helps to get rid more noise (high frequencies), however it leads to reduce the level of details [8], [10].The template weighting functions of averaging operator are unity (or 1/9 such as to guarantee that the outcome of averaging nine white pixels is white) as indicated in Figure 1.

Gaussian Averaging Operator
The Gaussian averaging operator is one of the best smoothing technique for noise removal of the image.The Gaussian relationship sets up the values of Gaussian operator template.As shown in (1), the formula describes the Gaussian function at coordinates (, ) and it is controlled by the variance To find the coefficients for Gaussian template using (1) which is then convolved with the image.The 3 × 3 convolution template for Gaussian with standard deviation σ = 1 is given in (2) [9], [11].

Edge Detection Techniques
The techniques of edge detection are based on mathematical methods which are capable of recognizing and detecting points in a digital image at which the brightness of the image is sharply changed.These critical points are typically arranged into groups of curved line segments called edges which are represented as binary image.The main objectives of edge detection are [9], [12]: a. Producing line-based drawings of scenes derived from their images.b.Extracting the important features from image edges such as corners, lines and curves.The algorithms of computer vision (e.g., recognition) are mostly utilized from such features.c.Intensity changes at different image locations are physically caused by various events such as [8], [13]; d. discontinuity in surface orientation and/or surface color and texture (surface boundary), e. discontinuity in depth and/or surface color and texture (object boundary), f. geometric and non-geometric events, g. direct reflection of light, such as a mirror (specularity), h.inter-reflections and shadows (from the same object or other objects).
The edge detection techniques comprise two groups; first order and second order.Examples of the first group are Robert, Sobel, Priwitt, kirsch and Robinson technique [14].Sobel is the most famous algorithm.Robinson technique is characterized by having many masks; as two of these masks are similar to those of Sobel technique.In the present work, Sobel and Robinson have been selected as representatives of first order group.On the other hand, the second order group includes mainly Laplacian, Laplacian of gaussain and Marr-Hildreth algorithm [15].The first two techniques are chosen to implement real time edge detection.

First Order Edge Detection
In image processing application, the calculation of the first derivative of a signal is approximated by finite differences as indicated in (3): if d=1, then the discrete derivative version is written as the local minima or maxima of first derivative are utilized to detect the points lying on an edge as illustrated in Figure 2 [8], [16].

Sobel Edge Detection Technique
The Sobel operator is based on central difference; however, it assigns higher weight value to the central pixels when averaging.The following equation gives 3×3 convolution of Sobel operators.
The two masks are individually applied on the input image to give gradient component in horizontal orientation G x and other gradient component in vertical orientation G y [3], [17].The magnitude gradient of is usually computed using (6): the gradient direction can be represented as:

b. Robinson Edge Detection Technique
The masks of Robinson edge detection masks are called compass masks since they are defined by rotating a single mask to eight main compass orientations: north, west, south, east, north-west, north-east, south-west, and south-east.The masks are depicted in Equation below: The maximum value resulting from the convolution of each of the masks with the image is defined as the edge magnitude.The edge detection is found by the mask that produces the maximum magnitude.It is worth to note that the masks R 2 and R 4 are identical to the Sobel masks.Extraction of explicit information about edge in any direction can be found by rotating any of the edge detection masks in a manner like compass masks [4], [18].

Second-order Edge Detection
Points which lie on an edge can be detected by detecting the zero-crossing of the second derivative as shown in Figure 2. The second derivative is approximated by finite differences as depicted in (9) [19]: The Laplacian definition of an image defined by f(x, y) is a second order derivative given by (10): The approximation version of above equation in digital image is given to 4 and 8-connectivities as indicated (11) and (12), respectively [11], [20]: Lane detection system for day vision using altera DE2 (Amjad J. Humaidi) where z i is the value of i-th position of matrix element; i.e., for 3 × 3 mask, the following mask can be represented in terms of z i [9], [21]. [ Laplacian operator is noise susceptible.To decrease the noise sensitivity, Laplacian of Gaussian (LoG) operator can be used instead.LoG is a two-step algorithm; it firstly performs the Gaussian smoothing followed by Laplacian operation.Low susceptibility of this method to noise is due to noise reduction caused by Gaussian function and probability of detection of false edges is considerably minimized by resultant Laplacian mask.The LoG function for convolution is defined as follows [14], [22]:

Thresholding (Gray Scale Thresholding)
Thresholding is a popular technique used for segmentation of monochrome image.The image intensity defined by f(x, y) comprises light object and dark background.A threshold level T is designated to extract the objects from background and the thresholded image g(x, y) can be described using (15).The (15) is only suitable when the image has 2 dominant gray level ranges [3], [24], [25].

Video Signal Formatting for Lane Detection
In what follows, video signal formatting will be elaborated for lane detection.The data read directly from camera and acquired by composite cable is available in the ITU656 format as illustrated in Figure 4 [3].These data have to be firstly converted into standard format YUV 4:2:2, which is commonly referred to as YCbCr.This format, YCbCr, is a color display format and it is composed of three components; the luma component (Y), blue-difference component (Cb) and red-difference chroma component (Cr).This format is a way of encrypting RGB data and it does not represent an actual color space of its own [4].As soon as the data is changed to this format, the system will sample down the signal from 720 to 640 horizontal pixels.Then the converted data is fed into an SDRAM FIFO which works as a frame buffer.Another conversion process is performed for the output of the FIFO, which transform the data from the format YUV 4:2:2 to the format YUV 4:4:4.At last, newly formatted YCbCr data is once more converted to the standard 10-bit RGB format.
The RGB data is directly fed into VGA controller to be next displayed on standard VGA monitor.However, instead of going directly to the VGA controller, the RGB data are admitted to go through other modules; filtering and edge detection modules [13].The above discussion is illustrated in Figure 5 and the functionality description of different modules in this figure is summarized in Table 1.The extension (.v) indicates that the Verilog is the programming Languages that has written the code.ISSN: 1693-6930  Lane detection system for day vision using altera DE2 (Amjad J. Humaidi)

355
It is interesting for edge detection to vary the threshold level to display different amounts of detail on the VGA screen.There are eighteen switches resident on the DE2 board.Table 2 illustrates how they can be utilized for resetting, switching filter and for manually adjusting threshold level of edge detection.

Result
In this section the lane edge detction will be performed based on the the used version of DE2 board and the camera, the results can be devided into either day or night lane detection.As mentioned earlier, the camera Genx (GDV 720 ultra) has two standards; PAL and NTSC, meanwhile Altera DE2 starting with (11xx...) supports only NTSC standard.Also, this type of camera does not support night vision; and hence, the day vision is only considered in the present scenario.Original and gray scale images are displayed in Figure 6 and Figure 7, respectively.

Filtering
To reduce the noise; the gray scale image is then fed to two filters, gaussain and avarage filters.Figure 8 and Figure 9 shows the images resulting from avarage and gaussain filters, respectively.It is clear from these figures that the average filter has better filtering charcteristics than Gaussian filter.Therefore, the latter filter will be excluded and average filter will be mainly relied on for filtering purposes in all next simulations.In what follows, the filtered image, based on average filter, is allowed to be edge detected.Two types of edge detection techniques has been suggested; first and second order techniques.

First-order Edge Detection
For the first order type, two techniques has been selected, Sobel and Robinson; while for the other type, Laplacian and Laplacian of Gaussian (LoG) has been chosen.Figure 10 and Figure 11 shows the images coming out from Sobel and Robinson techniques, respectively.One can easily see that more details would appear in image resulting from Robinson than that resulting from Sobel method.This may be attributed to eight mask operator which supports Robinson technique and four mask operator which support the other method.

Second-order Edge Detection
Next, the second order type has been considered.The edge detected images from Laplacian and LoG is illustrated in Figure 12 and Figure 13, respectively.It is well-known that this second order type could not give better results since they are based on detection of edge location from its zero-crossing property and hence such type of edge detction techniques would be sensetive to noise.However, it is clear from these figures that LoG gives better edge detection features than its counterpart.The conclusion drawn from the above argument is that Sobel edge detection technique could give the best edge detection characteristics and it outperforms the others.

Thresholding
It is interesting to examine the effectiveness of the considered techniques against the change of threshold level.Three levels have been taken into account; low, medium and high.These three levels has been practically set-up and designated using the switches implanted on DE2 board; such that the level selection is performed manually.Table 3 shows the bands of considered levels for both Sobel and Robinson.It is worthy to notify that the level band and value changes according to edge detection technique, environment, type of camera, board version and even on orientation angle of camera.For the present scenario, the low, medium and high level values for Sobel edge detection technique are designated as Sobel SW [2] to SW [3] SW [4] to SW [5] SW [6] to SW [7] Robinson SW [2] to SW [5] SW [6] to SW [8] SW [9] to SW [10] "SW [3]", "SW [5]" and "SW [7]", respectively.On the other hand, the low, medium and high-level values for Robinson edge detection technique are designated as "SW [5]", "SW [8]" and "SW [10]", respectively.The switch SW [0] is preserved for admitting either edge detection or gray scale image and the switch SW [1] is dedicated for switching the filtering process.Figures 14-25 show the images after detection with both first order techniques and with different three threshold levels.The conclusion drawn from this observation is that increasing the threshold level would better enhance the edge detected image.The reason behind this enhancement is that raising the threshold level would discard the weak features of the image with low intensity.Table 4 displays the flow summary report that appeared after Quartus II 32-bit version 11.1 succeeding in compilation.
It is clear that Laplacian technique wins the minimum numbers of logic elements.However, one may deduce from the table that the number of logic elements depends on number  ISSN: 1693-6930 TELKOMNIKA Vol.17, No. 1, February 2019: 349-361 360 of masks, their size and the strategy of edge detection computing; where the Sobel technique has 2 masks with 3 × 3 size, Robinson has 8 masks with 3 × 3 size, Laplacian has 1 mask with 3 × 3 size and LoG has 1 mask with 5 × 5 size.As such, LoG has maximum number of logic elements due to having maximum number of masks with large mask size which in turn reserve higher number of line buffers.It is worthy to notify that the total number of logic elements fabricated inside FPGA under study is 33,216 total logic elements and Sobel technique, e.g., occupy 9% of the total logic elements.It is interesting to investigate the change of table summary after thresholding process.Table 5 lists the information about number of logic elements, combination functions and dedicated logic registers for all considered edge detection techniques.However, due to deleting and adding instructions at the thresholding process, there is a slight change in the number of logic elements depending on the number of these elements required modified instructions.

Conclusions
Based on the observations of the results, one may draw the following main conclusions.For the sake of clarity, the conclusion points can be classified according to lane detection and lane tracking scenarios.Two filters have been suggested for lane detection part; average filter and Gaussian filter.The results of present work have shown that the average filter has much better filtering characteristics than Gaussian one and for all considered situations.For our case, raising the threshold level successfully enhanced the image resulting from most edge detection techniques.This indicates that all unimportant image features, clutter and noise reside at low level intensity.The results have shown that Sobel edge detector has the best edge detection characteristics among the others.Also, the threshold level raising has better enhancement for the image based on Sobel technique than that resulting from other techniques.For current work, second order edge detection techniques failed for performing edge detection in most scenarios and for different threshold level.The number of logic elements occupied by lane detection process relies on the number of masks and mask size for specified edge detection algorithm.In terms of minimum number of logic elements, Laplacian technique comes the first, while Sobel has the second order.Due to failure of Laplacian detector in most situations, Sobel become the first candidate edge detector.

Figure 3 (Figure 3 .
Figure 3 (a) shows the three-dimension plot of LoG function which looks like a Mexican hat.The above equation tells that a wider convolution mask is required for higher value of σ.It is easy to find out that the first zero-crossing of the LoG function occurs at √2σ n .Two-dimensional cross section of the zero-crossing of the Mexican hat is illustrated in Figure 3 (b)[17],[23].In (14) utilizes 5 × 5 convolution mask to carry out LoG edge detector.

Figure 5 .
Figure 5. Block diagram of a lane detection

Figure 6 .
Figure 6.Original color image Figure 7. Gray scale image

Table 1 .
Summary of Modules

Table 2 .
Control Edge Detections in DE2 Cyclone II

Table 3 .
Mechanism of Adjusting the Threshold of Edge Detection

Table 4 .
Summary of Resources for DE2 Start with-11

Table 5 .
Summary of Resources after Thresholding Process