- Deep Learning based Text Detection Using OpenCV (C++/Python)
- Deep Learning based Text Detection Using OpenCV (C++/Python)
- Image Text Recognition in Python
- Deep Learning based Text Recognition (OCR) using Tesseract and OpenCV
text-detectionThe importance of image processing has increased a lot during the last years. Especially with the growing market of smart phones people has started producing a huge amount of photos and videos which are continuously streamed on social platforms. Thus the increased interest of the industry towards this kind of problems is completely justified. Machine learning obviously plays a very significant role in this field. Automatic text detection and character recognition is just an example. One can cite other sophisticated applications such as animal species or plants identification, human beings detection or, more in general, extraction of any kind of information of commercial use. This field has been object of very intensive study in the past decades. Actually, at present, the problem of character recognition from black and white documents is considered solved. It is pretty common practice to scan a sheet of paper and use some standard software to convert it to a text file. In any case those are easy cases. The image are gray scale, very good contrast, no specific issue in single character contour detection and little problems due to lighting or shadows. A completely different scenario starts being depicted when we deal with natural scenes. For example a photo taken by a twitter user and then posted on the social platform. In this case the problem has not been solved at all. Actually there are still quite big issues in processing this kind of images. Of course the target of my project is not to find a final solution to this kind of open problem but in any case it is still worth trying and practice with such a fascinating topic. The post is organized as follows:. As you can see together with text at the bottom the background image is quite complex and overwhelming. The quote and the name of the author are also printed in two different font size which adds some sort of additional challenge to the task. After having loaded the image, it needs to be preprocessed. Specifically it goes through the next two steps:. After image cleaning, object detection is performed. Contours are identified and a rectangle is drawn around objects candidates. The result of this process is the following figure. As you can see a lot of rectangles have been identified. After this the objects are converted to greyscale, resized to 20 X 20 pixels and then stacked into a 3D numpy array. The coordinates of each rectangle are also saved in order to be able to reconstruct the image afterwards. The result of this operations is showed in the following standard output and generated figure plotting random images from the text-candidates selected :. Here comes the interesting part. The challenge now is to detect which ones of the identified objects contain text in order to be able to classify it. I approached the problem in the following way: basically I have to train a model to make such a decision which means that first of all I need a proper dataset, consisting ideally of half images containing text and half not containing it. To do that I decided two merge two existing data sources:. The complete dataset was then composed of k images, properly labeled and randomly shuffled. Then I needed a model to perform the binary classification.
Deep Learning based Text Detection Using OpenCV (C++/Python)
It is capable of 1 running at near real-time at 13 FPS on p images and 2 obtains state-of-the-art text detection accuracy. To discover how to apply text detection with OpenCV, just keep reading! Detecting text in constrained, controlled environments can typically be accomplished by using heuristic-based approaches, such as exploiting gradient information or the fact that text is typically grouped into paragraphs and characters appear on a straight line. Natural scene text detection is different though — and much more challenging. Due to the proliferation of cheap digital cameras, and not to mention the fact that nearly every smartphone now has a camera, we need to be highly concerned with the conditions the image was captured under — and furthermore, what assumptions we can and cannot make. With the release of OpenCV 3. The EAST pipeline is capable of predicting words and lines of text at arbitrary orientations on p images, and furthermore, can run at 13 FPS, according to the authors. Perhaps most importantly, since the deep learning model is end-to-end, it is possible to sidestep computationally expensive sub-algorithms that other text detectors typically apply, including candidate aggregation and word partitioning. To build and train such a deep learning model, the EAST method utilizes novel, carefully designed loss functions. For more details on EAST, including architecture design and training methods, be sure to refer to the publication by the authors. You may wish to add your own images collected with your smartphone or ones you find online. If you have any improvements to the method please do feel free to share them in the comments below. Before we get started, I want to point out that you will need at least OpenCV 3. To begin, we import our required packages and modules on Lines In order to perform text detection using OpenCV and the EAST deep learning model, we need to extract the output feature maps of two layers:. We load the neural network into memory using cv2. We need to filter out weak text detections by ignoring areas that do not have sufficiently high probability Lines 82 and The EAST text detector naturally reduces volume size as the image passes through the network — our volume size is actually 4x smaller than our input image so we multiply by four to bring the coordinates back into respect of our original image. The final step is to apply non-maxima suppression to our bounding boxes to suppress weak overlapping bounding boxes and then display the resulting text predictions:. As I mentioned in the previous section, I could not use the non-maxima suppression in my OpenCV 4 install cv2. This scene contains a Spanish stop sign. As you can tell, EAST is quite accurate and relatively fast taking approximately 0. We begin by importing our packages. Everything else is the same as in the previous section. This function is used to extract:. Our command line arguments are parsed on Lines Our frame is resized, maintaining aspect ratio Line From there, we grab dimensions and compute the scaling ratios Lines We then resize the frame again must be a multiple of 32this time ignoring aspect ratio since we have stored the ratios for safe keeping Line By optimizing our for loops with Cythonwe should be able to increase the speed of our text detection pipeline. To download the source code to this tutorial, and start applying text detection to your own images, just enter your email address in the form below. Enter your email address below to get a. All too often I see developers, students, and researchers wasting their time, studying the wrong things, and generally struggling to get started with Computer Vision, Deep Learning, and OpenCV. I created this website to show you what I believe is the best possible way to get your start.
Deep Learning based Text Detection Using OpenCV (C++/Python)
We live in times when any organisation or company to scale and to stay relevant has to change how they look at technology and adapt to the changing landscapes swiftly. We already know how Google has digitized books. Or how Google earth is using NLP to identify addresses. Or how it is possible to read text in digital documents like invoices, legal paperwork, etc. This post is about Optical character recognition OCR for text recognition in natural scene images. We will learn about why it is a tough problem, approaches used to solve and the code that goes along with it. Have an OCR problem in mind? Want to digitize invoices, PDFs or number plates? Head over to Nanonets and build OCR models for free! And moreover searching something in a large non-digital document is not just time-consuming but also, it is likely for us to miss the information while scrolling the document manually. Lucky for us, computers are getting better everyday at doing the tasks humans thought only they could do, often performing better than us as well. Some of the applications are Passport recognition, automatic number plate recognition, converting handwritten texts to digital text, converting typed text to digital text, etc. Many OCR implementations were available even before the boom of deep learning in While it was popularly believed that OCR was a solved problem, OCR is still a challenging problem especially when text images are taken in an unconstrained environment. I am talking about complex backgrounds, noise, lightning, different font, and geometrical distortions in the image. We can generally divide these tasks into two categories:. Structured Text- Text in a typed document. In a standard background, proper row, standard font and mostly dense. Unstructured Text- Text at random places in a natural scene. Sparse text, no proper row structure, complex backgroundat random place in the image and no standard font. But these techniques didn't properly work for a natural scene, which is sparse and has different attributes than structured data. In this blog, we will be focusing more on unstructured text which is a more complex problem to solve. As we know in the deep learning world, there is no one solution which works for all. We will be seeing multiple approaches to solve the task at hand and will work through one approach among them. There are lots of datasets available in English but it's harder to find datasets for other languages. Different datasets present different tasks to be solved. Here are a few examples of datasets commonly used for machine learning OCR problems. The Street View House Numbers dataset contains digits for training, digits for testing, and additional as extra training data. The dataset includes 10 labels which are the digits The dataset has bounding boxes around each digit instead of having several images of digits like in MNIST.
Image Text Recognition in Python
In order to obtain the bounding box x, y -coordinates for an object in a image we need to instead apply object detection. Object detection can not only tell us what is in an image but also where the object is as well. When combined together these methods can be used for super fast, real-time object detection on resource constrained devices including the Raspberry Pi, smartphones, etc. This will enable us to pass input images through the network and obtain the output bounding box x, y - coordinates of each object in the image. SSDs, originally developed by Google, are a balance between the two. The algorithm is more straightforward and I would argue better explained in the original seminal paper than Faster R-CNNs. We can also enjoy a much faster FPS throughput than Girshick et al. To learn more about SSDs, please refer to Liu et al. When building object detection networks we normally use an existing network architecture, such as VGG or ResNet, and then use it inside the object detection pipeline. The problem is that these network architectures can be very large in the order of MB. Network architectures such as these are unsuitable for resource constrained devices due to their sheer size and resulting number of computations. Instead, we can use MobileNets Howard et al. MobileNets differ from traditional CNNs through the usage of depthwise separable convolution Figure 2 above. The problem is that we sacrifice accuracy — MobileNets are normally not as accurate as their larger big brothers…. For more details on MobileNets please see Howard et al. If we combine both the MobileNet architecture and the Single Shot Detector SSD framework, we arrive at a fast, efficient deep learning-based method to object detection. I urge you to start there while also supplying some query images of your own. On Lines 41 and 42 we set the input to the network and compute the forward pass for the input, storing the result as detections. Computing the forward pass and associated detections could take awhile depending on your model and input size, but for this example it will be relatively quick on most CPUs. We start by looping over our detections, keeping in mind that multiple objects can be detected in a single image. We also apply a check to the confidence i. If the confidence is high enough i. Then, we extract the x, y -coordinates of the box Line 58 which we will will use shortly for drawing a rectangle and displaying text. Using the label, we print it to the terminal Line 62followed by drawing a colored rectangle around the object using our previously extracted x, y -coordinates Lines 63 and We display the resulting output image to the screen until a key is pressed Lines 70 and To be notified when future blog posts such as the real-time object detection tutorial are published here on PyImageSearch, simply enter your email address in the form below. Enter your email address below to get a. All too often I see developers, students, and researchers wasting their time, studying the wrong things, and generally struggling to get started with Computer Vision, Deep Learning, and OpenCV. I created this website to show you what I believe is the best possible way to get your start.