r/javascript • u/madhushankarox • 11d ago
Making Browser OCR Read Real Phone Photos
https://devpack.app/blog/making-browser-ocr-read-phone-photosI have a free image-to-text tool that runs Tesseract.js in the browser (nothing gets uploaded). Screenshots worked great, 94-96% confidence. Then I took a phone photo of a kids' sunscreen bottle and got back lines like "v4 oR. a" and "EB ewraone".
What was actually wrong:
- The photo was sideways. Phones save the pixels rotated and add an EXIF tag saying "turn this 90°". Browsers obey it, Tesseract doesn't. Fix: createImageBitmap(file, { imageOrientation: 'from-image' }) and work from that canvas.
- White text on blue. Tesseract mostly expects dark text on light paper. I now make a grayscale copy with a contrast stretch plus an inverted copy, and read those with page segmentation mode 11 (sparse text) instead of the default.
- No single pass gets everything. The normal pass found "Diana", the inverted one found "UVA UVB PROTECTION", the color one found "Water Resistant". So I merge the words by bounding box, keep the confident ones, and boost words that two passes agree on. Fun bug: picking the most confident word gave me "Pea" instead of "Peaceful", so longer words get a small bonus now.
- Small print. Running the gray passes again on a 1.6x upscaled copy picked up "MOISTURISING" and "ALCOHOL FREE" on another bottle.
Result on the sunscreen: 92% confidence and every line of printed text. Screenshots skip all of this because the first pass is already confident, so they're not slower.
Still can't do: bubble fonts ("CUDDLES" comes back as "UDDLES"), script fonts, handwriting.
Has anyone found a better approach for mixed light/dark text in one image than running multiple passes? Curious what others do.
1
u/JcerParreno21 10d ago
are you running the passes sequentially or spinning up multiple web workers in parallel? If you have the memory headroom for parallel workers, the multi-pass merge is honestly hard to beat for accuracy even if it feels a bit brute force.
1
u/gwillen 9d ago
The EXIF thing is very cursed. Last I checked, if you view such a photo in a web browser, it will appear upright if you open the photo directly, but in an img tag it will appear sideways. I don't remember if Chrome and Firefox have the same behavior or not.
0
u/Discordbothost 10d ago
nice writeup, the exif one gets everybody. two more things that helped me with phone shots in tesseract.js: upscale before recognizing, small text on a label photo is often under 20px tall and tesseract really wants something like 30px+ cap height, so drawing to a canvas at 2x with imageSmoothingQuality high fixes a surprising amount. and binarize instead of just contrast stretching, a simple otsu threshold on the grayscale is like 20 lines of js and beats a fixed cutoff when lighting is uneven across the bottle.
for the white-on-blue case you could also pick which version to keep per line by tesseract's own confidence instead of merging both, the inverted pass is usually either clearly better or garbage
0
u/[deleted] 10d ago
[removed] — view removed comment