Zero-Shot Tokenizer Transfer
Paper • 2405.07883 • Published • 5
How to use benjamin/zett-hypernetwork-multilingual-Mistral-7B-v0.1 with Transformers:
# Use a pipeline as a high-level helper
from transformers import pipeline
pipe = pipeline("feature-extraction", model="benjamin/zett-hypernetwork-multilingual-Mistral-7B-v0.1", trust_remote_code=True) # Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("benjamin/zett-hypernetwork-multilingual-Mistral-7B-v0.1", trust_remote_code=True, device_map="auto")Hypernetwork for Zero-Shot Tokenizer Transfer, see https://github.com/bminixhofer/zett