Design and Implementation of a Field Programmable Gate Array-Based Pedestrian Detection Framework for Autonomous Driving Application

Summary

This research article demonstrates the implementation of a real-time pedestrian detection algorithm on field-programmable gate array hardware, primarily for autonomous driving applications. The algorithm combines the histogram of oriented gradients (HoG) with a support vector machine (SVM) classifier, and results show efficiency in terms of speed, power consumption, and resource utilization.

Abstract

Autonomous driving offers a promising way to tackle the rising number of fatalities from traffic accidents. An autonomous vehicle includes many features, but the ability to detect pedestrians is crucial, challenging, and relevant to various real-time situations like surveillance, tracking people, and monitoring. Accurately identifying pedestrians is difficult because they can appear in different shapes, positions, and postures. They can wear various types of clothing and sometimes be partially hidden or blend in with nearby objects. This paper focuses on the real-time detection of pedestrians for self-driving cars using a popular hardware platform: The field programmable gate array (FPGA), Ultra 96 v2. The study implements a method for pedestrian detection based on a histogram of oriented gradients (HOG) combined with a support vector machine (SVM) classifier to recognize individuals on the FPGA board, leveraging high-level synthesis (HLS) tools. The effectiveness of the system has been tested on both still images and live video. The results show that advanced FPGA boards like the Ultra 96 v2 significantly improve performance metrics. The system operates at a clock frequency of 150 MHz while using less than half of the available resources and consuming around 2.5 W of power. Also, the system reports the pedestrian detection accuracy close to 95% and other efficient metrics for detection evaluation, like precision (78.6%), recall (88.3%), and F1 Score (83.1%). In summary, the developed system can detect pedestrians in real-time and has the potential to significantly improve the development of a smart and safe transportation environment.

Introduction

Urban development and the emergence of smart cities are topics of interest worldwide. All the nations are working towards the development of their cities that can be built in such a way that they can be safe and comfortable for the people living in them^1,2. But at the present situation, it has been observed that as the population grows and road congestion increases, the rate of fatalities resulting from road accidents due to driving negligence and poor visibility issues is rising alarmingly. A promising solution to these issues is the emergence of autonomous vehicles worldwide, which has sparked innovation^1,2,3,4 and researchers are making efforts to develop fully autonomous vehicles that can enable passengers to relax without any concerns. The need for the development of autonomous vehicles stems from the fact that even experienced drivers may face stress, dilemmas, fatigue, or difficulties sensing their environment due to bad weather, and all these issues lead to road accidents. A self-driving vehicle is designed to avoid accidents during travel, optimize engine resource use, and comply with traffic laws, which will undoubtedly enhance transportation^2,4. An autonomous vehicle is equipped with multiple features, sensors, and functionalities that allow it to sense its surroundings very precisely, avoiding any collisions and accidents, and hence they have now emerged as a promising solution for making the transport safe and secure^1,2,3,4.

Among all the features incorporated into an autonomous vehicle, one of the most vital is pedestrian detection. A robust pedestrian detection system can significantly help to lower road accident fatalities^5,6,7,8, as the majority of the victims of these accidents are pedestrians. Pedestrian detection involves identifying individuals on the road and avoiding any collisions with them. This feature is not only beneficial for self-driving cars, but also for various other application areas such as crowd monitoring, person identification, and tracking^9,10,11. The key aspect of this detection process is the speed and accuracy of detection. It is crucial to detect pedestrians accurately and quickly so that response time is minimal. There is an enormous challenge posed by pedestrian detection. Pedestrians on the road can be in any clothing, appearance, or posture, and may be invisible due to bad weather or occlusion^{10,11,12,13,14}. Moreover, it is quite possible that pedestrians might not follow the rules themselves, and one cannot control human nature, so the best possible approach is to equip the vehicle with the intelligence to handle any wrong action and avoid fatalities. The complete flow of the work in this research and the motivation behind the hardware implementation of pedestrian detection can be easily understood from Figure 1 below, which explains the need for pedestrian detection, its various application areas, the challenges involved, and the implementation on an FPGA to utilize the offered benefits.

Figure 1: Pedestrian detection. The need for pedestrian detection, key application areas of pedestrian detection, the challenges involved in pedestrian detection, and the implementation flow of pedestrian detection on an FPGA board. Please click here to view a larger version of this figure.

To identify pedestrians on the road, numerous algorithms exist. This overall task can be primarily divided into two main subtasks: the initial step extracts features from an input image, retaining only those that are significant and convey relevant information, while ignoring the redundant ones. For effective recognition of humans or pedestrians in an image, it is crucial that these features indicate the presence of a human figure within the scene^13,14. Following this extraction, the features need to be sent to a classifier that can determine whether the identified features correspond to a human. Therefore, the algorithm requires a feature extraction and description phase, followed by a classification step to determine pedestrian presence in the input scene. There are various algorithms available for this purpose. However, the most widely accepted method for pedestrian detection to date remains the combination of the histogram of oriented gradients (HoG) with the support vector machine (SVM) classifier^12,13,14,15. Numerous examples of software advancements exist, but ultimately, the goal is to port the implementation to a compatible hardware platform, which can then be integrated into the application system for real-time use. Therefore, the current emphasis is on hardware realization. It can thus be stated that it is required to develop a suitable hardware implementation of pedestrian detection systems, where cameras equipped with appropriate hardware can be deployed on vehicles and identify pedestrians on the road. When considering which hardware is appropriate for such implementations, one of the most commonly utilized options is the field programmable gate array (FPGA) due to its numerous advantages, including reduced design time, scalability, ease of modifications, reconfigurability, and lower energy and power consumption^{15,16,17,18,19,20,21,22}.

FPGA boards have consistently evolved and are now widely used for complex, advanced computer vision applications that span from basic image processing to object detection, augmented reality, and deep learning^20,21,22. Presently, several high-performance FPGA boards offer exceptional architectural capabilities to accommodate the extensive processing required for these intricate applications. If the advanced features of autonomous vehicles, such as pedestrian detection, are implemented on such hardware platforms, these platforms can be very useful for quick prototype development to analyze performance, and, after optimization, the implemented algorithm can be transferred to actual integrated circuits for integration into the system.

For over a decade, there have been significant publications based on the implementation of pedestrian detection using the HoG and SVM method on different FPGA platforms. Table 1 summarizes the articles in this field during the time from 2015–2025^{15,16,17,18,19,20,21,22,23,24}, focusing on the key factors like the image resolution, the throughput, or the frames per second (FPS), the type of classifier, and the key highlights or the contributions made by the paper.

Reference	FPGA Platform	Image Resolution	Classifier	Key Highlights / Contributions
15	Xilinx Zynq	640×480	AdaBoost	Real-time FPGA implementation; resource-efficient; uses binarization for optimization; good detection accuracy.
16	Terasic’s DE1-SOC board	640×480	SVM	High-performance HOG extractor; integrates SVM; single-scale detection; low-latency pipeline.
17	Altera DE2-115	640×480	AdaBoost	Evaluates performance at multiple viewpoints; FPGA implementation of HOG+AdaBoost; real-time pedestrian detection.
18	Intel Stratix V	640×480	SVM	Multi-scale pedestrian detection; FPGA-friendly HOG+SVM pipeline; highlights trade-offs between accuracy and hardware efficiency.
19	Zynq UltraScale+ MPSoC	3840×2160	SVM	Real-time UHD processing; pipelined HOG+SVM; SoC FPGA implementation; fixed-point optimization; scalable architecture.
20	Not specified	Not specified	SVM	Achieves >95% detection accuracy; real-time FPGA implementation; leverages parallelism; detailed HOG+SVM FPGA design for pedestrian detection.
21	Zync 7000 FPGA	1920×1080	SVM	High-throughput stream architecture for HOG+SVM; supports HD resolution; efficient pipeline for FPGA acceleration.
22	Ultra96 (rev1)	240×320	SVM	FPGA implementation using HLS; detects red traffic signals; calculates probabilities in 891 regions; latency ranges from 153,838 to 19 cycles.
23	Xilinx Zynq-7000 FPGA	640 × 480	HOG + SVM	Implemented pedestrian detection using HOG-SVM on FPGA, achieving real-time performance with reduced power consumption compared to CPU processing. Demonstrated optimized feature extraction pipeline suitable for embedded vision applications.
24	Xilinx Virtex-6 FPGA	640 × 480	Fixed-point object detector (Haar-like features)	Proposed high-throughput FPGA acceleration of object detection using fixed-point arithmetic, reducing computational cost while maintaining accuracy. Showed 15× speedup over CPU implementations with efficient hardware resource utilization.

Table 1: Literature review of research based on pedestrian detection on FPGA (2015–2025).

Table 1 summarizes that there has been extensive literature in the domain of pedestrian detection, and hardware implementation is the area of interest for researchers. It is also evident that there are advanced deep learning and machine learning techniques, such as convolutional neural networks (CNNs) based detectors like YOLO, transformer-based architectures, etc., for the task of pedestrian detection. They even outperform compared to the traditional HoG algorithm in terms of accuracy but when hardware implementation is considered then the advanced algorithms leads to huge resource utilization^23,24 due to the complexities of the algorithm which may also affect other performance parameters and it has also been observed that due to the added complexities the speed is slightly better in the case of the traditional HoG algorithm^24,25. It has also been observed that the advanced techniques consume more power when implemented on hardware^24,26. Thus, the aim of the work implemented in this paper is to perform pedestrian detection using the traditional HoG and SVM framework on FPGA hardware and achieve a favorable accuracy-speed-resource and power trade-off for real-time embedding. From Table 1 it is quite clear that when the HoG and SVM based work are analyzed then it is observed that there are limited publications that have utilized the recently introduced Zynq UltraScale+ MPSoC (Multi-Processor System On Chip) based FPGA development board²⁷ to explore the capability of these boards as from the point of view of architecture these boards have evolved, and offer great potential in terms of implementation of high end real-time computer vision application. There are limited publications that have realized the entire pedestrian detection system in real-time on the FPGA boards. However, they have focused on the efficient implementation or improvement of the intermediate tasks. Moreover, most of these implementations are based on realizing the entire system on an FPGA board through the use of hardware description languages. Few have utilized the benefits of High Level Synthesis (HLS) tools to speed up the design cycle. This paper demonstrates the design and implementation of real-time pedestrian detection on an FPGA board dedicated to an autonomous driving application. The paper utilizes the HoG and SVM framework for the purpose of pedestrian detection on still images, video, or real camera input. The hardware utilized is a cutting-edge and recently released FPGA board, the Ultra96 v2, which is an advanced FPGA architecture that is a powerful platform for computer vision, image processing, machine learning, edge computing, etc.²⁴. The Ultra96 v2 is a development board featuring an Arm-based AMD Xilinx Zynq UltraScale+ MPSoC²⁷. This board includes the processing system (PS) segment, which consists of ARM-based CPU cores that manage the software aspects of the project, and the Programmable Logic (PL) segment that allows for customizable hardware acceleration^20,21,22. Together, these components enhance the functionality of a hybrid system, where the PS part manages control and interaction with external elements, while the PL part handles the actual processing logic.

Protocol

The implementation procedure used in this research, based on pedestrian detection with HoG + SVM on an FPGA board, leveraging the benefits of high-level synthesis, is illustrated in Figure 2 below.

Figure 2: Design procedure for implementation of pedestrian detection on FPGA board. Phase 1: Pedestrian detection algorithm using HoG+SVM on HLS tool and generation of IP block. Phase 2: Pedestrian detection algorithm using HoG+SVM for actual FPGA implementation and generating the bit file. Phase 3: Programming the board with the generated bit file. Please click here to view a larger version of this figure.

1. Pedestrian detection using HoG and SVM on the HLS tool

1. Download the Python Integrated Development Environment (IDE) depending on the system properties on which the download is done.
   NOTE: In this work, Python 3.10 version is used.
2. Execute a Python script for running a training model using the HoG algorithm and the SVM classifier. Begin the script with loading the positive and the negative samples from the dataset.
   NOTE: Use the INRIA dataset¹¹.
3. Extract the HoG features for a window size of 64 x 128.
4. Split the training and the testing data from the INRIA dataset to a ratio of 80/20 with random shuffling for reproducibility.
   NOTE: The dataset is randomly shuffled using the python function before splitting the data. To ensure reproducibility so that the same splitting is done every-time a seed value is fixed in the code.
5. Train using the C-support vector classification (SVC) SVM with a linear kernel.
6. Extract the weight vectors and the bias.
7. Save the SVM weights and the bias in fixed-point Q8.8 format for the FPGA implementation.
   NOTE: The conversion is done by scaling each floating point values originality generated by a factor of 256 (2⁸) and casting the result to an integer.
8. Test the trained model through another Python script and adjust the regularization parameter C, until the accuracy is calculated to be more than 95%.
   NOTE: Optimized regularization parameter obtained: C = 0.05.
9. Open the HLS tool and create a new project with the part number selected as xczu3eg-sbva484-1-e.
10. Write a pedestrian detection code on the high-level synthesis tool using high-level languages like C++.
11. In the code, write three different C++ scripts: one for the HoG feature descriptor and the SVM classifier, another for the testbench to provide input to the test images and save the output images, and the third for a header file that declares the parameters used in the code.
12. In the code for HoG feature calculation, resize the image to 640 x 480 and apply a sliding-window architecture with a window size of 64 x 128. For every window, calculate the gradient magnitude and orientation for every overlapping 8 x 8 block.
    NOTE: It is important to scan the entire image through the sliding window concept so that every area of the image is covered and pedestrians of every possible size are identified.
13. In the other part of the same code for the HoG feature descriptor, pass the calculated gradients to the SVM classifier. Write the code for matching the features with the classifier weights and compare with the threshold to classify the detected feature as that of a human or not.
14. Click on Run C Simulation in the HLS tool to simulate the code with the help of a test bench to check the functionality correctness of the code.
15. Provide different input images to the code and check the output images with the detected pedestrians.
16. Click on Run C Synthesis to synthesize the code to map the code to hardware languages by the tool and generate the reports of timing and utilization.
    NOTE: The tool automatically opens the HLS synthesis reports. This report shows the estimate of the required clock frequency for implementing the coded task on selected FPGA platform, and also provides an estimate of the utilized resources. These reported values are only estimates and the actual parameters are calculated only after implementation on FPGA board.
17. Click on Export RTL to export the Intellectual Property (IP) block for the HoG algorithm of pedestrian detection.
    NOTE: This IP is to be used in the later stages of implementation.

2. Programming the FPGA board

1. Open the tool for FPGA programming and create a new project. Select the part number as xczu3eg-sbva484-1-e and create a new block design.
   NOTE: This block diagram is created to establish integration between the PS and the PL part of the FPGA board. The communication protocol used is the Advanced eXtensible Interface (AXI) protocol.
2. Search for the IP Catalog in the tool and once found open it.
3. Create a user repository by adding the path of the exported RTL IP in the step 1.17.
4. In the new block design window, now right click and select add IP. All the IPs will be visible including those that are provided by the tool as well as the user added repositories.
5. Add the Zync Ultrascale PS block from the repository.
   NOTE: This block reflects the PS part of the system, which is responsible for generating the required clocks, and it also has the master and slave ports for connection to the imported HoG IP via the AXI interconnect block, which runs on the AXI protocol fundamentals.
6. Add 8 HoG IPs because the system will be processing 8 windows simultaneously to leverage the benefit of parallelism offered by the FPGA board.
7. Add a processor system reset block, also that controls the clock and reset supplies to each and every block in the diagram.
8. Add two axi_smart connect blocks for connecting the HoG IPs with the Zync PS block. The entire block diagram with the complete connections is shown in Figure 3.
   NOTE: As shown in Figure 3, all the blocks that are required to be added to the design have been shown clearly. Figure 3 is captured from the tool and it shows the internal ports of every block as well as the interconnections between the ports of different blocks. This block is the main design as it is responsible for establishing the interface between the PS and the PL part of the FPGA board.
9. After completing the connections as per Figure 3, click on validate design.
   NOTE: Validation basically checks the missing connections or broken connections, which may lead to issues in later stages.
10. Click on synthesis and then Implement design in the tool after the validation of the block diagram is successful.
    NOTE: Synthesis maps the designed block diagram and maps it on the FPGA board. This step will report any violations that may indicate that the design cannot be implemented on hardware.
11. The tool will also generate reports related to timing, resource utilization, and power consumption. Carefully examine the reports to check for any timing violations and analyze the performance of the designed system.
12. Click on Generate the bitstream file which will generate the .bit file required for programming the FPGA board.

Figure 3: Block diagram for pedestrian detection using HoG + SVM imported IP. Please click here to view a larger version of this figure.

3. Final implementation on the FPGA board

1. Get the FPGA board and insert the SD card with the proper image file in the slot on the board.
2. Connect the board to the computer as well.
3. Boot the FPGA board in the SD card mode to enable Python programming on it^21,22.
4. Access the jupyter platform on the board after connecting it to the wifi.
5. Connect a web camera to the board.
6. Write a python code to import the generated bit file and perform the task of accessing the camera images.
7. In the code, write the script so that the image is written in the memory of the FPGA board through the PS part and passed onto the PL part for processing.
   NOTE: The PL part of the board that corresponds to the HoG IPs accesses the image pixels through these memory locations and processes them and provides the scores as output.
8. Write appropriate code in the same Python script to read the processed images and display them on the computer screen.
   NOTE: This completes the entire design and the system is now ready for deployment in real-world applications. All the codes used in this study are uploaded as supplemental coding files (Supplementary File 1 [Script_1_train_test.py], Supplementary File 2 [Script_2_HLS_hog.cpp], Supplementary File 3 [Script_3_HLS_test_bench.cpp], Supplementary File 4 [Script_4_HLS_consts.h], Supplementary File 5 [Script_5_jupyter_code.txt]).

Results

Pedestrian detection implementation on HLS
Figure 4 shows the simulation results on the HLS tool for the pedestrian detection using HoG + SVM. An input image with a pedestrian is fed as the test input to the code, and the output with the detected pedestrians is displayed. There are two sections in the image. The first detection has many bounding boxes around the same pedestrian again and again, and in the second image, the overlapping boxes are removed, and they are supp...

Discussion

This study effectively implements a real-time pedestrian detection system utilizing the HoG + SVM algorithm on advanced FPGA hardware based on Zynq UltraScale+ MPSoC development board²⁴. The results indicate that the traditional HoG algorithm of human detection¹¹ achieves accuracy close to 95% and utilizes only half of the on-board FPGA resources (LUTs, FFs, BRAM, DSPs), leaving huge capability of including much more processing to do some different tasks. When the entire im...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Python	Python	Version 3.10
Ultra 96 V2 FPGA Board	Xilinx	Introduced in 2018	Hardware Implementation Platform used for implementing the pedestrian detection algorithm
Vivado	AMD	2019.2	FPGA Programming tool used for programming the Ultra 96 v2 FPGA board with the pedestrian detection algorithm
Vivado HLS	AMD	2019.2	High Level Synthesis Tool used for high level programming of the pedestrian detection code in the paper to export the Intellectual Property (IP)