Hello, I am trying to build a Neural Network on Xilinx Spartan 3e xc3s500e, that I will feed it with images from this camera: OV7670 and train it in order to determine if the person in the camera is man or woman. Is there any VHDL core capable of implementing this processing? Ideally I would try to achieve real-time (delay = 1-2 sec) to decide if the picture shows man or woman. The camera I have bought is the OV7670, which takes 640x480 pictures. Can it work or I need a megapixel camera? Thank you...

I don’t know much about image processing with neural networks, but I’m curious as to how the network is going to be interfaced. I started work on a similar neural network so I find this topic interesting. I was planning on detecting images from 32x32 bitmaps. I’m assuming there’s a small number of neurons that are being recycled multiple times and this is mainly a software task? I would think the camera resolution would be sufficient (based on the opinion that most people could distinguish men from women at that resolution) but getting the processing done in time might be a challenge. The FPGA doesn’t have an integrated processor so some sort of softcore is likely involved. With a 640x480 camera there are 307,200 inputs minimum to be processed by the network. I would try and estimate the required bandwidths in the system. Maybe setup in a spreadsheet so that arguments can be varied. I’m guessing the camera resolution is more than what can be processed in real-time.

Thank you. Something that I cannot understand right now and I want to ask it, is this:
If I manage to build a neural network, and, for example, give it pictures with caucasian faces to output if it sees woman or man, will it work if I use images with Chinese (for example) faces to output if it sees woman or man? I mean can I use the same Core to train it with different images? Thank you...

It won't happen with that FPGA (at least not an ideal, dynamic, self-learning system). There are multiple steps involved with what you are trying to do. It's not as simple as feed it a picture, process the image and presto it spits out if it is a man or women. You have to answer questions like

1. Will this be in grayscale or full color (3x the data if in full color, but possibly missing subtle differences if only in grayscale)
2. How will the data be presented to the system
3. Will this be a controlled environment (picture booth, good lighting) or uncontrolled (outside video, cloudy/rainy days)
4. Is the subject allowed to have makeup on, scarfs, glasses, hats, eyes open/closed, etc
5. Will the subject be viewed from different angles or always the same
6. Can subjects be men dressed as women or women dressed as men (or ??) and the system is still supposed to differentiate

You have to know the above because it dictates the overall complexity of this design and how feasible it will be.

Getting past the picture representation (i.e. controlled vs uncontrolled, grayscale vs. color, lighting, etc), one of the first problems will be recognizing what part of the picture is actually a face. From there you will probably have to normalize it. Or basically translate/rotate it (i.e. if the person is looking 7 degrees slanted/rotated vs. the last time at 0 degrees straight on). And then scale it (i.e. if the person happens to be 5 feet closer to the camera then last time at 6 feet). There will probably be a couple more steps before you can even process it with the neural network. What this basically means is if you don't do data reduction/preprocessing it will require a MASSIVELY powerful FPGA to process the amount of data required in real time and if you do the data reduction/preprocessing...well, it will still require a powerful FPGA to process it all in real time. What you have probably wouldn't cut it (at least not by itself). It would require doing quite a bit of data reduction and preprocessing (or shortcuts) that would eventually lead to reduced accuracy and a non-viable solution.

If you are serious about this you will need to obtain a dataset that will be used to train and test your setup (unless you happen to have 500+ men and 500+ women of varying age, height, weight, etc. willing to take a few snapshots for this). You will need to make sure the dataset is representative of the environment you are expecting the video feed to be in if you expect the system to work well with out-of-sample tests. And you will also need a means to feed your setup with this dataset (might mean specialty hardware just for this purpose alone). Using some for just training and some for just testing...and a few out-of-sample to test the final solutions. It's possible you can do this in software and then transfer the final neural network weights to the FPGA saving you from trying to do it via your hardware setup. Lookup how to train and test a neural network to understand this better...

I would highly recommend you also lookup 'face detection' and 'facial recognition' just to see what is involved and what is already available before going one step further (never reinvent the wheel). If you are willing to use multiple FPGAs or more powerful FPGAs (or maybe just a static, hard-coded solution) this could be done, but there will still be quite a bit of work involved to get there...

1. Will this be in grayscale or full color (3x the data if in full color, but possibly missing subtle differences if only in grayscale)
---> Color
2. How will the data be presented to the system
---> What do you mean? The output will be the string "man" or "woman".
3. Will this be a controlled environment (picture booth, good lighting) or uncontrolled (outside video, cloudy/rainy days)
---> Good lighting
4. Is the subject allowed to have makeup on, scarfs, glasses, hats, eyes open/closed, etc
---> "Clean" face in front of the camera. No glasses, no hats,not eyes closed.
5. Will the subject be viewed from different angles or always the same
---> No, the face will be in front of the camera.
6. Can subjects be men dressed as women or women dressed as men (or ??) and the system is still supposed to differentiate
---> No, assume men dressed like men, women dressed like women.

I have built the system on Raspberry Pi 3 B+, in python, but it takes 30-35 sec to decide if the person is man or woman. It is far to slow... I am trying to accelerate that as far as I can.

What I am also looking is the solution of PYNQ boards that are more flexible (support both python and VHDL and have many interfaces) or Virtex 4 (or 5) which I find on ebay for up to 150 euros. I cannot spend more that 120 - 150 euros. What is your opinion about these boards for what I am trying to build? Thank you for your time...

I would recommend this board https://www.ebay.com/itm/182512610736 for development. Final FPGA model you can always reduce if you find out you don't need this much (and this board is in your budget - the FPGA it uses retails for over 2200.00 Euros but the board sells for 111.35 Euros). If you do need an embedded MCU though, you might need to find an FX version (vs. the LX version) which has a built in PowerPC 405 core(s) with higher performance then the LX. This LX model can still do multiple soft cores, it just won't have the performance of the FX version and if you are trying to do this in real time that could be the difference. Otherwise, for performance and price...you can't beat it.

Since you've already developed a working model on the Pi, use that as a means for initiating an interface and the VHDL code required. While running the existing system, time bits/segments of your code. Take the first bit/segment that takes the longest and figure out how to pipe it through the FPGA. Then take the second bit and so on until you've either achieved the performance needed or have packaged it entirely in the FPGA. Then at that point you'd proceed to optimize the VHDL code.

Have you timed bits/segments of your existing system to determine what's the bottleneck? If you have a static neural net that will be the easiest to implement into the FPGA since then it is a matter of data piping and parallel execution. Do you have a trained/working neural network that performs with a high enough precision? Or is this going to be a dynamic system that will learn as it goes?

Hello,

I have build the neural network and is trained with caucasian people and it outputs the string: "man" or "woman". When I use chinese people, again it is trained on this new dataset and outputs the string "man" or "woman". So it is dynamic neural network. I cannot make it decide in time less than 30 sec. That is why I am trying building it on FPGA.

Thank you

I have a slightly different recommendation:

1) Don't buy dev hardware from ebay...

2) Don't buy a board that uses a Xilinx Virtex-4... You'll be stuck using a fairly dated version of ISE (I actually like ISE for it's simplicity, but it's getting pretty long in the tooth these days...) The 7-series parts are MUCH nicer to work with...

3) Consider, for example, Trenz Electronics vs. Ebay... You can pick up an Artix-7 dev board for about the same price as the ebay board, and Trenz stands behind their products... The 7-series, even in the Artix line, is a pretty big jump over the Virtex-4...

Here's an example: https://shop.trenz-electronic.de/en/TE0710-02-35-2IF-Dual-fast-Ethernet-Artix-Module-with-Xilinx-Artix-7-35T-ind.-temp.-range

This part is supported by Vivado WebPack as well, so no need to buy a Vivado license... I'm not so sure about the XC4VLX100...

Just some thoughts...

/jw

jw, You're right, if he doesn't have a license then ISE won't support the XC4VLX100 (and this would go way over his budget). The Artix-7 board you point out is a nice board...but only 1 piece of a set. Buying that and the base (or even just a cable to fit into it's interfaces) will also put him over his budget of 120 - 150 euros. So if he doesn't have a license and is interested in the Artix-7, then I would recommend https://www.ebay.com/itm/173712711664 (~52.52 Euros) if you want SDRAM and https://www.ebay.com/itm/173165541550 (~66.55 Euros) if you want DDR3 RAM. Both well within his budget and both supported by Vivado WebPack. And yes it's eBay again...its just for development (so it needs to be cost effective and functional).

Have you timed subsections of the code yet? In general, a neural network (NN) that is trained shouldn't take that long to process data...to train on it, yes. But just to process it to produce an output no. Have you reviewed your code to make sure when it is testing, it isn't still training?

You will probably still need to time your code. Time the section that grabs the data from the camera into data/picture blocks itself (if you find this is your bottleneck you'll be wasting your time on an FPGA NN - part of why this timing thing is so important before you even start). Time the section that processes the data before presenting it to the NN (I'd hope you don't present the data to the NN in raw format - otherwise this will be the cause of the bottleneck that will result in the NN processing time being extreme). Time the section of the NN processing itself. Break it down further if you get sections that use up a large percentage of the total processing. You probably don't need to time the NN output section since that should be taking only a few milliseconds. Then ratio your times out to the total processing time and let us know your results.

I'm not quite sure what you mean by "one piece of a set"... The board I sent a URL for is a standalone board. He can pick up a carrierboard if he wants, but otherwise, the TE0710 is very usable standalone. I've used this board's siblings, standalone, many times... He'll have to do a little work to wire up a DC supply, but this represents a Saturday afternoon at best... The biggest two advantages are:

1) He'll actually get what he paid for...
2) He'll get well crafted documentation (including full schematics).

I don't have a dog in this race, but if the OP is actually trying to get work done, vs. tinkering around, then he'll probably want to purchase from a known good vendor vs. purchasing from someone on e-bay. The OP has a problem with the board, Trenz will stand behind it... The OP needs support... Trenz will help... You gets what you pays for...

Here's one more difference... The ebay board has a 6-pin JTAG connector... The Trenz board has a dual JTAG/UART interface via USB that works out of the box with Vivado... With the e-bay board, assuming he doesn't have a JTAG cable, he'll need to add another 60-ish USD for something like a Digilent HS2 JTAG cable (or he'll have to take a Saturday to build the equivalent).

/jw

But, Virtex 4 has 110,592 Logic Cells and Artix7 has 33,280 logic cells. Does not this impact on how much bigger circuit I can build on VHDL?

Yes... Taken simply, it does... However...

You can't use that Virtex 4 without an ISE license. I'm pretty sure Vivado doesn't support Virtex 4 (confirmed with Vivado 2018.2)... The free version of ISE (WebPack), won't support that particular device. ISE has been deprecated for about 5 years or so... I think you'd have to buy a Vivado license to make use of licensed ISE. If you are a member of a University Community of some kind, you may be able to get 'hold of a Vivado license. If not, it's fairly expensive (~3,000 USD, give or take). The Artix-7 (35T) works with Vivado Webpack

Also, comparing 4-series and 7-series architectures is more complicated than just counting LCs... Vivado, in my opinion (at least from about 2017.1+), does a much more efficient job of routing than ISE ever did. There are also some pretty big differences between the two hardware architectures. Xilinx provides details on both architectures on-line.

In the end, what you need is going to be fairly application dependent. If you purchase the Virtex 4 board, understand that you won't get support if you have a hardware problem (e.g. the board is DOA), you will need an ISE license, and you'll also need a JTAG cable/interface of some kind. Have you considered starting with a more accessible board? (or maybe a simulator?)

Best!

/jw

The Digilent CMOD A7 comes with the A35 device and is in a 40pin DIL form. Cheap at ~£70 from Farnells in the UK. This is perfect for bread-boarding a design. It has ~36 pins which can be used for I/O. It has a built in USB device to program the bitstream from the free Vivado WebPack software.

So how about partitioning a design across 2 or more devices (if the design will not fit in a given FPGA). It means you will have to think about how you can split up the proposed design to fit, but you could use a Verilog/VHDL simulator to verify the combined design and also the partitioned sub-designs.

Divide and Conquer!

Regards,
JD McMullin, PhD

I have finally bought Virtex 5. What better camera can I use instead of OV7670 I have posted in previous post? In the same FPGA I have to put, also, the core of the neural network and the core of the serial interface, which will output man or woman. I am thinking of RNN..