Video Processing on the FPGA of a Zybo using VHDL

(Last Updated On: March 10, 2018)

Processing video frames require a large amount of computation power, therefore processors may not be powerful enough. In these cases FPGAs provide a good solution allowing the implementation of parallel modules, which will process all the information a lot faster than a processor does.

C:\Users\Alberto\AppData\Local\Microsoft\Windows\INetCache\Content.Word\Schema-video-processing.png

Introduction

The main topic of this article is how to process video using a FPGA with VHDL as a hardware description language. The target board is in this case ZYBO board from Digilent based on the Xilinx SoC Zynq family z7010.

The main function is to create a small functional project, documented and with some application examples, where other users or students can add easily their own VHDL code using Vivado HLS for create a video processing final application.

In the appendix, a small tutorial is inserted explaining how to use Vivado from the beginning until the creation of the video processing application.

Video processing in the Zybo board

The Zybo board have one HDMI and one VGA port. Each of these video connectors could be used as a sink or as a source, in other words input or output. In this project, the HDMI will be used as an input because almost all the normal photo or action cameras they have an HDMI output, that can be used conveniently for this purpose. Another Digilent Zynq-based development boards may have different sink/source video connectors. For example, the Nexys Video Artix-7 FPGA or Arty Z7, they have 2 HDMI port, one always configured as an input and the other one as an output.
HDMI–>; Video input
VGA –>; Video output

C:\Users\Alberto\AppData\Local\Microsoft\Windows\INetCache\Content.Word\Schema-video-processing.png
Connection block of the system

In this project only the FPGA part of the Zynq SoC will be used due to the aim of this studying subject to develop a VHDL program. Also using the FPGA, a much more higher data throughput than using the microprocessor can be reached, therefor higher video processing speed. The processor system (PS) of the Zynq chip could also be used to process this video by software. In the next table the most important pros and cons are resumed:

FPGA processing with VHDL Software processing with ARM processor
Low abstraction level video processing Low, mid or high abstraction level video processing
High speed and parallel computing High frame rate and resolution possible Processing speed limited by the processor and possible bottle neck on the processor low frame rate and resolution

The IPs (Intellectual Property) used in this project are provided by Digilent and the last version can be found on the official Github repository. These IPs code and decode from/to different video signal protocols. In this case, a conversion from HDMI to RGB raw video signal is needed at the input and a conversion from RGB video raw video signal to VGA at the output port. Of course these block modules could be done in VHDL by the user, but it would take lots of time, and we don’t need to reinvent the wheel.

The clock signal source requested by the HDMI to RGB converter block is 200 MHz (Data sheet [1] page 5). This signal is given by the processor clock.

The camera to use can be any camera with an HDMI output. Nowadays, almost every cameras have this output. This code was tested with one “action camera” and a ‘normal’ camera Sony Cibershot.

Examples of camera, which will work as a signal input

https://pevly.com/wp-content/uploads/2016/01/Eken-H9-action-camera.jpg

 

Vivado block design

The block diagram shown in the Figure 3 was made to interconnect different modules. The following main IPs were inserted:

  • Dvi2rgb: It make a conversion from the video input of the HDMI into 24-bits RGB video. From Digilent library [2]. When the HDMI port is operating as a sink, the DDC function is needed for the connected source to read out the characteristics of the device. [3]
  • Rgb2vga: it converts the raw input video signal into VGA output. From Digilent library [2].
  • Processing_system7: This is a special block, that contains the configuration of the processing system of the Zynq. For this application, the outpu clock from the processing unit is going to be used.
  • VideoProcessing: A user defined RTL module. Here the user-application code for video processing should be inserted and modified.
Minimal block diagram

Additional to these blocks two constants are used to configure the HDMI port as a sink. [3]

Block diagram adding the buttons, sliders and leds of the ZYBO board

To ease the user interface and to include some complementary options, the sliders, buttons and leds available on the ZYBO board are routed to the application block. This allows a better functionality and choices to program the final video processing application. The block diagram adding these features is shown in the Figure 4:

RGB video Signals: a small background

To process video signals there are many protocols and codification technics. The aim of this program is to use raw video data because it is easy to handle and to understand.

Therefore the HDMI (High Definition Multimedia Interface) input will converted into rgb 24 bit video signal, known as “RGB24”. In the same way the modified rgb video output is converted into VGA video signal.

RGB24 video enconding can provide 16 million of distinct colors. A synchronization signal is necessary to display the long strings of pixels properly on the screen. The video decoders block implemented on the design, they use the called RGBHV synchronization. (RGBHV stands for Red Green Blue Horizontal Vertical)

This method of transmitting video (RGBHV) adds a horizontal synchronization video timing signal (pHSync) and another for vertical synchronization (pVSync). Both signals are independent of itself and of the colors, having in total 5 signals to transmit as shown in Figure 1. Besides a video data valid signal indicates if the video is active or is a blanking period and a pixel clock recovered from the input TDMS clock channel.

RGB video output

Video Processing user application with VHDL

A RTL module was created and instanciated on the main block design. Inside this RTL module the user can add and edit HDL-language code. This could be done with Verilog or VHDL language by changing the module properties.

To rapid edit the RTL module content, by pressing F7 after selecting the RTL block on the design. Also it can be done by opening it from the source window.

The easiest application would be to drive straight the video signal through the FPGA and not modify it. This code will be used for testing the connections and as well the camera output signal. The example program can be considered as a “hello world” application. The code can be found on the Code 1.

Code : Drive through VHDL application

Black and white or Grayscale video effect

This block aims to create a grayscale video output from a color RGB video input. For this purpose, the red, green and blue channel need to be unified to form a unique color pixel for the three output channels. There are diverse ways of weighted sums, in this project the NTSC Rec.601 was employed, because is one of the most digital standard definition formats. This is due the human eye is more sensitive to green than red or blue. This formula try to compensate this human eye effect. [4]

Therefore, the equation is:

Y = 0.299·Red + 0.587·Green + 0.114·Blue

Weighted grayscale of the colors:

Red 29.9% 0x4C = 76

Green 58.7% 0x97 = 151

Blue 14.4% 0x1C = 28

Sum = 76 + 151 + 28 = 255

Extract for the algorithm for grayscale video converting

VideoEdition:

The output of this code can be seen on the Figure 6. There a normal camera, on the lower right corner, is pointing to the blue bottle.

C:\Users\Alberto\AppData\Local\Microsoft\Windows\INetCache\Content.Word\DSC00604.jpg
Shade of Gray video processing

Pin configuration

The pin assignment is made after the synthesis is run correctly. This process can be done with the Vivado I/O configuration tool (see Figure 7), but this task could be tedious if the number of pins to configure is large. In this case is more practical to do it by tcl instructions placed on the constraint file, due the total pins to configure is at least 15 for the HDMI and 18 for the VGA.

Vivado I/O configuration tool

The configuration of the pins was made consulting the ZYBO schematic [5]. The I/O port definition of this project is stored on the file pin_descriptions.xdc

Alternatives and other libraries

This project was an example of using HDL to video processing, but for further development and high-end solutions may be consider the use of already on the market tools and libraries. Here two interesting tools are presented:

  • Xilinx video and image processing library called ”Video Processing Subsystem” [6]. This library developed by Xilinx can help and ease to create video processing algorithms with useful IPs for Vivado.
  • Matlab HDL Video toolbox [7]. This Matlab tool generate a HDL code using the technique of Model Based Design.

References

[1]Digilent, «DVI-to-RGB (Sink) 1.6 IP Core User Guide,» 2016.

[2]Digilent, «Digilent ZYBO Video IP Repository,» Mar 2016.

[3]Digilent, «ZYBO Reference Manual,» 2016.

[4]A. R. Adrian Ford, «Colour Space Conversions» 1988.

[5]Digilent, «Zybo Schematic».

[6]Xilinx, «Video Processing Subsystem Library pack» .

[7]Matlab, «Vision HDL Toolbox» .

15 thoughts on “Video Processing on the FPGA of a Zybo using VHDL

  1. Reply
    Ben Ma - May 14, 2018

    Hi Alberto, I got VHDL code error when I try to do the code for grayscale. signal red, green, blue: std_logic_vector(7 downto 0); signal grayscale: std_logic_vector(23 downto 0); begin red <= vid_data(23 downto 16); green <= vid_data(15 downto 8); blue <= vid_data(7 downto 0); process(clk_pix) begin if rising_edge(clk_pix) then if sliders(0) = '1' then grayscale <= ((X"4C" * red) + (X"97" * green) + (X"1C" * blue)); –weighted color end if; end if; end process; –video signals OUT_vid_data(23 downto 16) <= grayscale(15 downto 8); –red channel OUT_vid_data(15 downto 8) <= grayscale(15 downto 8); –green channel OUT_vid_data(7 downto 0) <= grayscale(15 downto 8); –blue channel Vivado shows there are errors for: grayscale <= ((X"4C" * red) + (X"97" * green) + (X"1C" * blue)); –weighted color part. errors: find '0' definition for operator "*", cannot determine exact overloaded matching definition for "*"; find '0' definition for operator "+", cannot determine exact overloaded matching definition for "+". Could you help me to see what wrong for the Vhdl code?

    1. Reply
      Alberto L. - May 14, 2018

      Hello, it seems that the math library is missing.
      Can you try with:
      library IEEE;
      use IEEE.STD_LOGIC_1164.ALL;

      at the beginning
      Regards. Alberto

  2. Reply
    ShiHyun Ahn - May 15, 2018

    Hello , I am a researcher(south korea) and a begginer with video processing on FPGA. Your Mis Circuitos give me many godd information. I tested your “Video Processing on the FPGA of a Zybo using VHDL” with Gopro. but I continuously failed to test “HDMI(input) – VGA(output)” on Vivado 2016.4 I think cause resolution in the default EDID only support 3 resolutions (https://forum.digilentinc.com/topic/991-zybo-hdmi-sink/) I test again with modifing the file dgl_dvi_edid.txt in DVI-to-RGB (Sink) 1.6 IP. but failed like link https://forum.digilentinc.com/topic/3483-issues-with-dvi2rgbedid-files-on-zybo/ How can I set EDID resoultion in in DVI-to-RGB (Sink) 1.6 IP? If you share your “Video Processing on the FPGA of a Zybo using VHDL” code(included XDC, video library IP, Project file..), It is best way to resolve this problem. I am trying to test this during 2 months. Please help me~ Issues with dvi2rgbEDID files on Zybo (https://forum.digilentinc.com/topic/3483-issues-with-dvi2rgbedid-files-on-zybo/)

  3. Reply
    Alberto L. - May 16, 2018

    Hello ShiHyun,
    the versions, and updates of the IPs can cause that codes can be obsolete and need some update. I dindt try this project for a year.
    I will try to find if I have my original project and share with you.
    I hope you the best with your projects!
    best Regards. Alberto

  4. Reply
    Bambi Stroope - June 2, 2018

    I’m truly enjoying the design and layout of your website. It’s a very easy on the eyes which makes it much more pleasant for me to come here and visit more often. Did you hire out a developer to create your theme? Great work!

    1. Reply
      Alberto L. - June 3, 2018

      Thanks for your comment. No, I develop it by myself…
      Regards

  5. Reply
    best strap - June 27, 2018

    I am so grateful for your post.Much thanks again. Really Cool.

  6. Reply
    Stimulator - June 27, 2018

    Pretty! This was an incredibly wonderful article. Thanks for supplying this information.

  7. Reply
    Maria - June 27, 2018

    Very neat article. Keep writing.

  8. Reply
    Andrew - June 28, 2018

    “Thank you ever so for you blog.Thanks Again.”

  9. Reply
    John - June 28, 2018

    wow, awesome blog post.Thanks Again. Much obliged.

  10. Reply
    toenail fungus - June 29, 2018

    I really like and appreciate your article.Thanks Again. Much obliged.

  11. Reply
    didillop - June 30, 2018

    Appreciate you sharing, great blog post.Much thanks again. Awesome.

  12. Reply
    Reformas vitoria - June 30, 2018

    I think this is among the most vital info for me. And i am glad reading your article. But should remark on some general things, The web site style is great, the articles is really great
    D. Good job, cheers

  13. Reply
    plug - July 2, 2018

    Thanks so much for the article post.Much thanks again.

Leave a Reply

Your email address will not be published. Required fields are marked *

Scroll to top