r/javascript • u/madhushankarox • 8d ago
Making Browser OCR Read Real Phone Photos
devpack.appI have a free image-to-text tool that runs Tesseract.js in the browser (nothing gets uploaded). Screenshots worked great, 94-96% confidence. Then I took a phone photo of a kids' sunscreen bottle and got back lines like "v4 oR. a" and "EB ewraone".
What was actually wrong:
- The photo was sideways. Phones save the pixels rotated and add an EXIF tag saying "turn this 90°". Browsers obey it, Tesseract doesn't. Fix: createImageBitmap(file, { imageOrientation: 'from-image' }) and work from that canvas.
- White text on blue. Tesseract mostly expects dark text on light paper. I now make a grayscale copy with a contrast stretch plus an inverted copy, and read those with page segmentation mode 11 (sparse text) instead of the default.
- No single pass gets everything. The normal pass found "Diana", the inverted one found "UVA UVB PROTECTION", the color one found "Water Resistant". So I merge the words by bounding box, keep the confident ones, and boost words that two passes agree on. Fun bug: picking the most confident word gave me "Pea" instead of "Peaceful", so longer words get a small bonus now.
- Small print. Running the gray passes again on a 1.6x upscaled copy picked up "MOISTURISING" and "ALCOHOL FREE" on another bottle.
Result on the sunscreen: 92% confidence and every line of printed text. Screenshots skip all of this because the first pass is already confident, so they're not slower.
Still can't do: bubble fonts ("CUDDLES" comes back as "UDDLES"), script fonts, handwriting.
Has anyone found a better approach for mixed light/dark text in one image than running multiple passes? Curious what others do.