Pathumma-llm-vision-2.0.0-preview

Model Overview

Pathumma-llm-vision-2.0.0-preview is a multi-modal language model fine-tuned for Visual Question Answering (VQA) and Image Captioning tasks. It contains 8 billion parameters and leverages both image and text processing to understand and generate multi-modal content.

Model Name: Pathumma-llm-vision-2.0.0-preview
Base Model: Qwen/Qwen2-VL-7B-Instruct
Architecture: Multi-modal LLM (Visual Language Model)
Parameters: 7 Billion
Organization: NECTEC
License: [Specify License]

Intended Use

Primary Use Cases:
- Visual Question Answering (VQA)
- Image Captioning
Intended Users: Developers, researchers, and AI practitioners working on multi-modal tasks.
Possible Applications: Educational tools, accessibility applications, interactive visual content generation.

Model Description

Pathumma-llm-vision-2.0.0-preview is designed to perform multi-modal tasks by integrating both visual and textual information. The model is fine-tuned with diverse datasets to improve its ability to understand and generate content that aligns with both image and text inputs.

Training Data

The model was fine-tuned on several datasets:

Thai Image Caption: Data sourced from image captioning competitions on Kaggle.
Small-Thai-Wikipedia: Articles in Thai from Wikipedia.

Dataset Size

Training Dataset Size: 132,946 examples
Validation Dataset Size: - examples

Training Details

Hardware Used:
- HPC Cluster: Lanta
- Number of Nodes: 4 Nodes
- GPUs per Node: 4 GPUs
- Total GPUs Used: 16 GPUs
Fine-tuning Duration: 20 hours, 34 minutes, and 43 seconds (excluding evaluation)

Evaluation Results

Type	Encoder	Decoder	IPU24-dataset (test) (Sentence SacreBLEU)
Pathumma-llm-vision-beta-0.0.0	siglip-so400m-patch14-384	Meta-Llama-3.1-8B-Instruct	13.45412
Pathumma-llm-vision-1.0.0	siglip-so400m-patch14-384	Meta-Llama-3.1-8B-Instruct	17.66370
Pathumma-llm-vision-2.0.0-preview	Qwen2-VL-7B-Instruct	Qwen2-VL-7B-Instruct	19.112962

**Note: Other models not target fine-tuned on IPU24-datasets may be less representative of IPU24 performance.

Required Libraries

Before you start, ensure you have the following libraries installed:

pip install transformers==4.48.1 accelerate peft bitsandbytes qwen-vl-utils[decord]==0.0.8

Usage

We provide a inference tutorial. To use the model with the Hugging Face transformers library:

import torch

from peft import get_peft_model, LoraConfig
from transformers import BitsAndBytesConfig
from transformers import (
    Qwen2VLForConditionalGeneration,
    Qwen2VLProcessor,
)

MODEL_ID = "nectec/Pathumma-llm-vision-2.0.0-preview"
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
USE_QLORA = True

lora_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.05,
    r=8,
    bias="none",
    target_modules=["q_proj", "v_proj"],
    task_type="CAUSAL_LM",
)

if USE_QLORA:
    bnb_config = BitsAndBytesConfig(
        load_in_8bit=True,
        # load_in_4bit=True,
        # bnb_4bit_use_double_quant=True,
        # bnb_4bit_quant_type="nf4",
        # bnb_4bit_compute_type=torch.bfloat16
    )


model = Qwen2VLForConditionalGeneration.from_pretrained(
    MODEL_ID,
    device_map="auto",
    quantization_config=bnb_config if USE_QLORA else None,
    torch_dtype=torch.bfloat16
)


model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

MIN_PIXELS = 256 * 28 * 28
MAX_PIXELS = 1280 * 28 * 28

processor = Qwen2VLProcessor.from_pretrained(MODEL_ID, min_pixels=MIN_PIXELS, max_pixels=MAX_PIXELS)

def encode_via_processor(image, instruction, question):

    if isinstance(image, str):
        local_path = image
        image = Image.open(local_path)

    messages = [
            {
                "role": "system", "content": [{"type": "text", "text": instruction}]
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "image"
                    },
                    {
                        "type": "text",
                        "text": question
                    }
                ]
            },
        ]

    text = processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
    ).strip()

    def convert_img(image):
        width, height = image.size
        factor = processor.image_processor.patch_size * processor.image_processor.merge_size
        if width < factor:
            image = image.copy().resize((factor, factor * height // width))
        elif height < factor:
            image = image.copy().resize((factor * width // height, factor))
        return image
    image_inputs = [convert_img(image)]

    encoding = processor(
        text=text,
        images=image_inputs,
        videos=None,
        return_tensors="pt",
    )

    ## Remove batch dimension
    # encoding = {k:v.squeeze(dim=0) for k,v in encoding.items()}
    encoding = {k: v.to(DEVICE) for k, v in encoding.items()}
    inputs = encoding
    return inputs


def encode_via_processor_extlib(local_path, instruction, question):
    img_path = "file://" + local_path
    messages = [
            {
                "role": "system", "content": [{"type": "text", "text": instruction}]
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "image": img_path,
                    },
                    {
                        "type": "text",
                        "text": question
                    }
                ]
            },
        ]

    text = processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
    ).strip()

    image_inputs, video_inputs = process_vision_info(messages)

    encoding = processor(
        text=text,
        images=image_inputs,
        videos=video_inputs,
        return_tensors="pt",
    )

    ## Remove batch dimension
    # encoding = {k:v.squeeze(dim=0) for k,v in encoding.items()}
    encoding = {k: v.to(DEVICE) for k, v in encoding.items()}
    inputs = encoding
    return inputs

def inference(inputs):
    start_time = time.time()
    model.eval()
    with torch.inference_mode():
        # Generate
        generated_ids = model.generate(
            **inputs,
            max_new_tokens=256,
            temperature=.1,
            # repetition_penalty=1.2,
            # top_k=2,
            # top_p=1,
        )
        generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)
    end_time = time.time()

    ## Get letency_time...
    latency_time = end_time - start_time

    answer_prompt = [*map(
        lambda x: re.sub(r"assistant(:|\n)?", "<||SEP-ASSIST||>", x).split('<||SEP-ASSIST||>')[-1].strip(),
        generated_texts
    )]
    predict_output = generated_texts[0]
    response = re.sub(r"assistant(:|\n)?", "<||SEP-ASSIST||>", predict_output).split('<||SEP-ASSIST||>')[-1].strip()

    return predict_output, response, round(latency_time, 3)

instruction = "You are a helpful assistant."

def response_image(img_path, question, instruction=instruction):
    image = Image.open(img_path)
    _, response, latency_time = inference(encode_via_processor(image=image, instruction=instruction, question=question))
    print("RESPONSE".center(60, "="))
    print(response)
    print(latency_time, "sec.")
    print("IMAGE".center(60, "="))
    plt.imshow(image)
    plt.show()

# Output processing (depends on task requirements)
question = "อธิบายภาพนี้"
img_path = "/content/The Most Beautiful Public High School in Every State in America.jpg"
response_image(img_path, question)

>>> ==========================RESPONSE==========================
>>> อาคารสีน้ำตาลขนาดใหญ่ที่มีเสาไฟฟ้าอยู่ด้านหน้าและมีต้นไม้อยู่ด้านข้าง
>>> 7.987 sec.
>>> ===========================IMAGE============================
>>> <IMAGE_MATPLOTLIB>

Limitations and Biases

The model may exhibit biases due to the training data, which might not be fully representative of all contexts.
Performance may degrade on unfamiliar images or non-standard question formats.

Ethical Considerations

The model should not be used to generate misleading information or in ways that violate privacy.
Consider fairness and minimize bias when using the model for language and image processing tasks.

Citation

If you use this model, please cite it as follows:

@misc{PathummaVision,
  author = {Thirawarit Pitiphiphat and NECTEC Team},
  title = {nectec/Pathumma-llm-vision-2.0.0-preview},
  year = {2025},
  url = {https://huggingface.co/nectec/Pathumma-llm-vision-2.0.0-preview}
}

@article{Qwen2VL,
  title={Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution},
  author={Wang, Peng and Bai, Shuai and Tan, Sinan and Wang, Shijie and Fan, Zhihao and Bai, Jinze and Chen, Keqin and Liu, Xuejing and Wang, Jialin and Ge, Wenbin and Fan, Yang and Dang, Kai and Du, Mengfei and Ren, Xuancheng and Men, Rui and Liu, Dayiheng and Zhou, Chang and Zhou, Jingren and Lin, Junyang},
  journal={arXiv preprint arXiv:2409.12191},
  year={2024}
}

@article{Qwen-VL,
  title={Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond},
  author={Bai, Jinze and Bai, Shuai and Yang, Shusheng and Wang, Shijie and Tan, Sinan and Wang, Peng and Lin, Junyang and Zhou, Chang and Zhou, Jingren},
  journal={arXiv preprint arXiv:2308.12966},
  year={2023}
}

Contributor Contract

Vision Team
Thirawarit Pitiphiphat (thirawarit.pit@ncr.nstda.or.th)
Theerasit Issaranon (theerasit.issaranon@nectec.or.th)

Contact

For questions or support, please contact https://discord.gg/3WJwJjZt7r.

This formatting provides a clean, structured, and readable Markdown layout for these sections. Let me know if further adjustments are needed!

nectec
/

Pathumma-llm-vision-2.0.0-preview