Ultrasound scanning has been used as the preliminary diagnosis tool all over the world. The ultrasound data is being analyzed by Tele-radiologists. It lacks online availability. The drawbacks of Tele-radiology have been overcome by using Computer Aided Diagnosis. As an aid to this, the Random Forest Classifier has been used here for detecting kidney abnormalities. The initial Pre-Processing stage filters the speckle noise existing in the input kidney image. Then feature extraction has been performed. Image categorization as normal, cyst and stone has been done with Random Forest Classifier. Then the performance is evaluated by comparing it with K-nearest neighbor classifier and Support Vector Machine. From experimentation, it is observed that the accuracy and F-measure values of Random Forest Classifier ranges high when compared with other classifiers due to the continuous split of the trees until accurate categorization is done. Simulation and Implementation has been done using Modelsim 6.4a and Xilinx Spartan-6 FPGA board.