Repository navigation
feat(mistral): extract PDF images as files and expose table format in Mistral Parser #9856
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+432
−131
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
58bdd83
feat(mistral): extract PDF images as files and expose table format in…
waleedlatif1 82d4147
fix(mistral): type the v3 images output as optional for the shared v2…
waleedlatif1 0f67286
fix(mistral): name extracted images as storage does, drop inline tabl…
waleedlatif1 e8cf711
docs(mistral): describe Extract Images and Tables options in the Mist…
waleedlatif1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| import { describe, expect, it } from 'vitest' | ||
| import { extractOcrImageFiles, MAX_OCR_EXTRACTED_IMAGES } from '@/lib/internal/mistral/images' | ||
|
|
||
| const JPEG = Buffer.concat([Buffer.from([0xff, 0xd8, 0xff]), Buffer.from('synthetic jpeg body')]) | ||
| const dataUri = (bytes: Buffer) => `data:image/jpeg;base64,${bytes.toString('base64')}` | ||
|
|
||
| function image(id: string, imageBase64: string | undefined) { | ||
| return { | ||
| id, | ||
| top_left_x: 0, | ||
| top_left_y: 0, | ||
| bottom_right_x: 10, | ||
| bottom_right_y: 10, | ||
| image_base64: imageBase64, | ||
| } | ||
| } | ||
|
|
||
| describe('extractOcrImageFiles', () => { | ||
| it('moves image bytes into files named as storage will name them, with no base64 left in the page output', () => { | ||
| const { output, files } = extractOcrImageFiles({ | ||
| pages: [ | ||
| { | ||
| index: 2, | ||
| markdown: '', | ||
| images: [image('img-0.jpeg', dataUri(JPEG))], | ||
| }, | ||
| ], | ||
| }) | ||
| expect(files).toEqual([{ buffer: JPEG, name: 'page-2-img-0.jpg', mimeType: 'image/jpeg' }]) | ||
| expect(JSON.stringify(output)).not.toContain(JPEG.toString('base64')) | ||
| expect(output.pages).toEqual([ | ||
| expect.objectContaining({ | ||
| images: [expect.objectContaining({ id: 'img-0.jpeg', file_name: 'page-2-img-0.jpg' })], | ||
| }), | ||
| ]) | ||
| }) | ||
|
|
||
| it('stores at most the image cap and keeps the bounding boxes of the rest without a file', () => { | ||
| const images = Array.from({ length: MAX_OCR_EXTRACTED_IMAGES + 1 }, (_, i) => | ||
| image(`img-${i}.jpeg`, JPEG.toString('base64')) | ||
| ) | ||
| const { output, files } = extractOcrImageFiles({ pages: [{ index: 0, images }] }) | ||
| expect(files).toHaveLength(MAX_OCR_EXTRACTED_IMAGES) | ||
| const pageImages = (output.pages as Array<{ images: Array<Record<string, unknown>> }>)[0].images | ||
| expect(pageImages).toHaveLength(MAX_OCR_EXTRACTED_IMAGES + 1) | ||
| expect(pageImages.at(-1)).not.toHaveProperty('file_name') | ||
| expect(pageImages.at(-1)).not.toHaveProperty('image_base64') | ||
| }) | ||
|
|
||
| it('skips an image over the per-image cap instead of failing the parse', () => { | ||
| const huge = Buffer.alloc(21 * 1024 * 1024) | ||
| const { output, files } = extractOcrImageFiles({ | ||
| pages: [ | ||
| { | ||
| index: 0, | ||
| images: [image('img-0.png', dataUri(huge)), image('img-1.jpg', dataUri(JPEG))], | ||
| }, | ||
| ], | ||
| }) | ||
| expect(files.map((file) => file.name)).toEqual(['page-0-img-1.jpg']) | ||
| expect(JSON.stringify(output)).not.toContain('image_base64') | ||
| }) | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| import { isRecordLike } from '@sim/utils/object' | ||
| import { decodeDataUriWithinLimit } from '@/lib/file-parsers/data-uri' | ||
| import { isFileParserError } from '@/lib/file-parsers/errors' | ||
| import type { InternalToolFile } from '@/lib/internal/tool-operations/file-result' | ||
| import { MAX_BUFFERED_TRANSFER_BYTES } from '@/lib/uploads/shared/types' | ||
| import { resolveEffectiveMimeType } from '@/lib/uploads/utils/file' | ||
| import { resolveStoredFileMetadata } from '@/lib/uploads/utils/stored-file-metadata' | ||
|
|
||
| /** Most images one parse stores; Mistral's `image_limit` is clamped to it. */ | ||
| export const MAX_OCR_EXTRACTED_IMAGES = 100 | ||
|
|
||
| /** Largest single image kept; a bigger one is a page scan, not a figure. */ | ||
| const MAX_OCR_IMAGE_BYTES = 20 * 1024 * 1024 | ||
|
|
||
| interface OcrImageFiles { | ||
| /** The OCR output with every `image_base64` removed and each kept image named. */ | ||
| output: Record<string, unknown> | ||
| files: InternalToolFile[] | ||
| } | ||
|
|
||
| /** The decoded image, or null when it is over the per-image cap. */ | ||
| function decodeImage(imageBase64: string): { buffer: Buffer; mediaType: string | null } | null { | ||
| const dataUri = imageBase64.startsWith('data:') ? imageBase64 : `data:;base64,${imageBase64}` | ||
| try { | ||
| return decodeDataUriWithinLimit(dataUri, MAX_OCR_IMAGE_BYTES) | ||
| } catch (error) { | ||
| if (isFileParserError(error) && error.code === 'complexity_limit') return null | ||
| throw error | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Turns the base64 images in a Mistral OCR response into files the executor | ||
| * stores, so image bytes never travel through block outputs. Each kept image | ||
| * gets a `file_name` matching its stored file; an image past the count, the | ||
| * per-image cap, or the total byte budget keeps its bounding box but no file. | ||
| */ | ||
| export function extractOcrImageFiles(output: unknown): OcrImageFiles { | ||
| if (!isRecordLike(output)) return { output: {}, files: [] } | ||
| if (!Array.isArray(output.pages)) return { output, files: [] } | ||
| const files: InternalToolFile[] = [] | ||
| let totalBytes = 0 | ||
| const pages = output.pages.map((page: unknown) => { | ||
| if (!isRecordLike(page) || !Array.isArray(page.images)) return page | ||
| const pageIndex = typeof page.index === 'number' ? page.index : 0 | ||
| const images = page.images.map((image: unknown) => { | ||
| if (!isRecordLike(image)) return image | ||
| const { image_base64: imageBase64, ...rest } = image | ||
| if (typeof imageBase64 !== 'string' || !imageBase64) return rest | ||
| if (files.length >= MAX_OCR_EXTRACTED_IMAGES) return rest | ||
| const decoded = decodeImage(imageBase64) | ||
| if (!decoded || totalBytes + decoded.buffer.length > MAX_BUFFERED_TRANSFER_BYTES) return rest | ||
| totalBytes += decoded.buffer.length | ||
| const id = typeof image.id === 'string' && image.id ? image.id : `img-${files.length}.jpeg` | ||
| /** Storage renames an image from its bytes; naming it the same way keeps `file_name` matching. */ | ||
| const draftName = `page-${pageIndex}-${id}` | ||
| const { fileName, mimeType } = resolveStoredFileMetadata( | ||
| draftName, | ||
| resolveEffectiveMimeType(decoded.mediaType, draftName), | ||
| decoded.buffer | ||
| ) | ||
| files.push({ buffer: decoded.buffer, name: fileName, mimeType }) | ||
| return { ...rest, file_name: fileName } | ||
| }) | ||
| return { ...page, images } | ||
| }) | ||
| return { output: { ...output, pages }, files } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.