Is there a OCR toolkit or a ML Model which is able to reliable extract tables from invoices?
By far the best one I've come across is Microsoft Azure Document Intelligence with the Layout Model[0].
It's really, really good at tables.
You have to use the Layout Model and not just the base Document Intelligence.
A bit pricey, but if you're processing content one time and it's high value (my use case as clinical trial protocol documents and the trial will run anywhere from 6-24 months), then it's worth it, IMO.
[0] https://learn.microsoft.com/en-us/azure/ai-services/document...
All frontier multi modal LLMs can do this - there’s likely something lighter weight as well.
In my experience, the latest Gemini is best at vision and OCR
> All frontier multi modal LLMs can do this
There's reliable, and there's reliable. For example [1] is a conversation where I ask ChatGPT 4o questions about a seven-page tabular PDF from [2] which contains a list of election polling stations.
The results are simultaneously impressive and unimpressive. The document contains some repeated addresses, and the LLM correctly identifies all 11 of them... then says it found ten.
It gracefully deals with the PDF table, and converts the all-caps input data into Title Case.
The table is split across multiple pages, and the title row repeats each time. It deals with that easily.
It correctly finds all five schools mentioned.
When asked to extract an address that isn't in the document it correctly refuses, instead of hallucinating an answer.
When asked to count churches, "Bunyan Baptist Church" gets missed out. Of two church halls, only one gets counted.
The "Friends Meeting House" also doesn't get counted, but arguably that's not a church even if it is a place of worship.
Longmeadow Evangelical Church has one address, three rows and two polling station numbers. When asked how many polling stations are in the table, the LLM counts that as two. A reasonable person might have expected one, two, three, or a warning. If I was writing an invoice parser, I would want this to be very predictable.
So, it's a mixed bag. I've certainly seen worse attempts at parsing a PDF.
[1] https://chatgpt.com/share/67812ad9-f2bc-8011-96be-faea40e48d... [2] https://www.stevenage.gov.uk/documents/elections/2024-pcc-el...
I wonder if performance would improve if you asked it to create csvs from the tables first, then fed the CSVs in to a new chat?
https://github.com/microsoft/table-transformer
This is much lighter weight and more reliable than vllm
Ah so like NIM is a set of microservices on top of various models, and this is another set of microservices using NIM microservices to do large scale OCR?
and that too integrated with prometheus, 160GB VRAM requirement and so on?
Looks like this is targeted for enterprises or maybe governments etc trying to digitalize at scale.
I have hard time to understand what they mean by "early access micro services"...?
Does it mean that it is yet another wrapper library to call they proprietary cloud api?
Or that when you have the specific access right, you can retrieve a proprietary docker image with secret proprietary binary stuffs inside that will be the server used by the library available in GitHub?
The latter. NIMs is Nvidia's umbrella branding for proprietary containerized AI models, which is being pushed hard by Jensen. They build models and containers, then push them to ngc.nvidia.com. They then provide reference architectures which rely on them. In this case the images are in an invite only org, so to use the helm chart you have to sign up, request access, then use an API key to pull the image.
You can imagine how fun it is to debug.
Nvidia getting in on the lucrative gpt-wrapper market.
lol, while checking which OCR is using (PaddleOCR) I found a line with the text: "TODO(Devin)" and was pretty excited thinking they were already using Devin AI...
"Devin Robison" is the author of the package!! Funny, guess it will be similar with the name Alexa
Sounds pretty useful. What are the system requirements?
Prerequisites
Hardware
GPU Family Memory # of GPUs (min.)
H100 SXM or PCIe 80GB 2
A100 SXM or PCIe 80GB 2
Seems pretty ridiculous to me to parse some PDFs. Almost like they made this as bloated as possible to justify buying $5,000+ GPUs for an office.
Wow, I perhaps need a kubernetes cluster just for a demo:
CONTAINER ID IMAGE
0f2f86615ea5 nvcr.io/ohlfw0olaadg/ea-participants/nv-ingest:24.10
de44122c6ddc otel/opentelemetry-collector-contrib:0.91.0
02c9ab8c6901 nvcr.io/ohlfw0olaadg/ea-participants/cached:0.2.0
d49369334398 nvcr.io/nim/nvidia/nv-embedqa-e5-v5:1.1.0
508715a24998 nvcr.io/ohlfw0olaadg/ea-participants/nv-yolox-structured-images-v1:0.2.0
5b7a174a0a85 nvcr.io/ohlfw0olaadg/ea-participants/deplot:1.0.0
430045f98c02 nvcr.io/ohlfw0olaadg/ea-participants/paddleocr:0.2.0
8e587b45821b grafana/grafana
aa2c0ec387e2 redis/redis-stack
bda9a2a9c8b5 openzipkin/zipkin
ac27e5297d57 prom/prometheus:latestAlso, they're rolling the dice continuing to use Redis https://github.com/redis/redis/blob/21aee83abdbfe8878d8b870b...
You can just use k3s/rke2 and run everything on the same node.
It may be least of your worries considering it requires 2x[A/H]100 80GB Ram.