tencent

/

EVIE-4.5B

Like 33

Follow

Tencent 12.5k

Hello @xiaoxiaoshadiao and Team!

Congratulations on the big non-preview release! SOTA once again! Like the Preview model, I also wanted to integrate these 2 with Sentence Transformers to give users more inference options. I recognize that you're still actively making changes, so just let me know if you need me to incorporate changes from main or something.

Heads up, this PR was AI-generated and human-reviewed.

Pull Request overview

• Integrate EVIE-4.5B with Sentence Transformers via MultiVectorEncoder
• Add usage examples for full-width embeddings and smaller Matryoshka prefixes

Details

This follows the Preview integration from https://huggingface.co/tencent/EVIE-Preview-4.5B/discussions/1. I've kept the ColPali usage first, as requested there, and added the Sentence Transformers example below it.

The integration uses Transformer -> Dense -> Normalize -> MultiVectorMask, with the trained projection weights and bias extracted from this checkpoint into 1_Dense/model.safetensors. Bidirectional attention is configured through sentence_bert_config.json, and the existing 768-token visual budget is preserved. A separate retrieval chat template reproduces the ColPali input format while keeping the original chat template unchanged. The processor metadata resolves the standard Qwen3VLProcessor, so no ColPali imports or trust_remote_code are needed.

The default output is 2048-dimensional. The README also shows how to obtain smaller Matryoshka prefixes by slicing and renormalizing each token embedding.

This PR is primarily additive. The only changes to existing runtime configuration are processor_class and padding_side. The ColPali path explicitly loads ColQwen3_5Processor and hardcodes left padding in its constructor, so it does not rely on these metadata values. The existing ColPali inference code and model configuration are unchanged, and ColPali behavior should therefore remain the same.

Verified against this repository's ColPali implementation in float32 with Transformers 5.15.0: full-width token embeddings are identical, and the maximum MaxSim difference across all six supported dimensions is 1.9e-6. The reference was also checked with the pinned Transformers 5.13.1, with MaxSim differences below 3.1e-5 between versions. The README output comes from a separate default-dtype run using the public image URLs below.

To try this integration before the PR is merged, use the PR revision below.

pip install -U "sentence-transformers[image]>=6.0.0"

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("tencent/EVIE-4.5B", revision="refs/pr/1")

queries = [
    "What is the variable represented on the y-axis of the graph?",
    "Total outlay is maximum in which year?",
]
documents = [
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc3.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc4.jpg",
]

query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# torch.Size([23, 2048]) torch.Size([755, 2048])

scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
# tensor([[17.0977,  7.8730,  6.9902,  4.8066],
#         [ 4.4795, 12.8516,  4.2031,  4.3779]])

Happy to adjust anything. Please let me know if you have any questions or feedback!

• Tom Aarsen