Can I run this model locally , if yes how?

#2
by Pramil120 - opened

want to rum this model locally on my pc, as new in the ai world how can I do that, please guide. Ollama / LMStudio not able to make it work.

I tried Kaggle and it is not working either. I am thinking to try locally too but did you have any luck so far ?

The reason is that AyurParam uses a custom architecture (ParamBharatGenForCausalLM) with trust_remote_code=True, and it hasn't been converted into the GGUF format that apps like LM Studio or Ollama require.

If you want to run it locally or in a Python environment, you cannot use the desktop app GUIs directly. Instead, you need to run it via Python using the Transformers library like this:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
"bharatgenai/AyurParam",
trust_remote_code=True,
device_map="auto"
)

Alternatively, you can serve it via vLLM or SGLang as shown in the model card instructions. Hope this helps clear up why it fails on standard local UI tools!

•
This comment has been hidden

Sign up or log in to comment