File size: 3,326 Bytes
2f9b4d1 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 |
---
base_model: Qwen/Qwen-VL-Chat
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** Agora Research
- **Model type:** Vision Language Model
- **Language(s) (NLP):** English/Chinese
- **Finetuned from model:** Qwen-VL-Chat
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** https://github.com/QwenLM/Qwen-VL
- **Paper:** https://arxiv.org/pdf/2308.12966.pdf
## Uses
```
from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers.generation import GenerationConfig
```
# Note: The default behavior now has injection attack prevention off.
```
tokenizer = AutoTokenizer.from_pretrained("qwen/Qwen-VL-Chat",trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
"MODEL_PATH_HERE", # path to the output directory
device_map="cuda",
trust_remote_code=True
).eval()
```
# Specify hyperparameters for generation (generation_config if transformers < 4.32.0)
```
#model.generation_config = GenerationConfig.from_pretrained("Qwen/Qwen-VL-Chat", trust_remote_code=True)
# 1st dialogue turn
query = tokenizer.from_list_format([
{'image': 'https://images.rawpixel.com/image_800/cHJpdmF0ZS9sci9pbWFnZXMvd2Vic2l0ZS8yMDIzLTA4L3Jhd3BpeGVsX29mZmljZV8xNV9waG90b19vZl9hX2RvZ19ydW5uaW5nX3dpdGhfb3duZXJfYXRfcGFya19lcF9mM2I3MDQyZC0zNWJlLTRlMTQtOGZhNy1kY2Q2OWQ1YzQzZjlfMi5qcGc.jpg'}, # Either a local path or an url
{'text': "What are the objects in the image? What animals are present? Are there any people in the image?"},
])
print("sending model to chat")
response, history = model.chat(tokenizer, query=query, history=None)
print(response)
```
# Print Results
```
[FUNCTION CALL]
{{
'type': 'object',
'properties': {{
'objects': {{
'type': 'array',
'description': 'The objects present in the image.',
'items': {{
'type': 'string',
'enum': ['dog', 'person', 'tree', 'path', 'sun']
}}
}},
'animals': {{
'type': 'array',
'description': 'The animals present in the image.',
'items': {{
'type': 'string',
'enum': ['dog']
}}
}},
'people': {{
'type': 'boolean',
'description': 'Whether there are people in the image.',
'enum': [true]
}}
}}
}}
[EXPECTED OUTPUT]
{{
'objects': ['dog', 'person', 'tree', 'path', 'sun'],
'animals': ['dog'],
'people': true
}}
```
### Direct Use
Just send an image and ask a question in the text.
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
(recommended) transformers >= 4.32.0
## How to Get Started with the Model
```
query = tokenizer.from_list_format([
{'image': 'https://images.rawpixel.com/image_800/cHJpdmF0ZS9sci9pbWFnZXMvd2Vic2l0ZS8yMDIzLTA4L3Jhd3BpeGVsX29mZmljZV8xNV9waG90b19vZl9hX2RvZ19ydW5uaW5nX3dpdGhfb3duZXJfYXRfcGFya19lcF9mM2I3MDQyZC0zNWJlLTRlMTQtOGZhNy1kY2Q2OWQ1YzQzZjlfMi5qcGc.jpg'}, # Either a local path or an url
{'text': "QUESTIONS/QUERIES GO HERE"},
])
```
## Training Details
### Training Data
Custom Function Calling Dataset with 70k examples
### Training Procedure
qlora for 3 epochs
|