File size: 3,106 Bytes
e56f5fb 9cb2773 23eb523 e56f5fb 9cb2773 8e18da9 9cb2773 b847b44 9cb2773 08cf992 9cb2773 7b3af89 9cb2773 23eb523 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 |
---
license: llama3.1
datasets:
- remyxai/vqasynth_spacellava
tags:
- remyx
---
# Model Card for SpaceLlama3.1
**SpaceLlama3.1** uses [llama3.1-8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B) as the llm backbone along with the fused DINOv2+SigLIP features of [prismatic-vlms](https://github.com/TRI-ML/prismatic-vlms).
## Model Details
Uses a full fine-tune on the [spacellava dataset](https://huggingface.co/datasets/remyxai/vqasynth_spacellava) designed with [VQASynth](https://github.com/remyxai/VQASynth/tree/main) to enhance spatial reasoning as in [SpatialVLM](https://spatial-vlm.github.io/).
### Model Description
This model uses data synthesis techniques and publically available models to reproduce the work described in SpatialVLM to enhance the spatial reasoning of multimodal models.
With a pipeline of expert models, we can infer spatial relationships between objects in a scene to create VQA dataset for spatial reasoning.
- **Developed by:** remyx.ai
- **Model type:** MultiModal Model, Vision Language Model, Prismatic-vlms, Llama 3.1
- **Finetuned from model:** Llama 3.1
### Model Sources
- **Dataset:** [SpaceLLaVA](https://huggingface.co/datasets/remyxai/vqasynth_spacellava)
- **Repository:** [VQASynth](https://github.com/remyxai/VQASynth/tree/main)
- **Paper:** [SpatialVLM](https://arxiv.org/abs/2401.12168)
## Usage
Try the `run_inference.py` script to run a quick test:
```bash
python run_inference.py --model_location remyxai/SpaceLlama3.1
--image_source "https://remyx.ai/assets/spatialvlm/warehouse_rgb.jpg"
--user_prompt "What is the distance between the man in the red hat and the pallet of boxes?"
```
## Deploy
Under the `docker` directory, you'll find a dockerized Triton Server for this model. Run the following:
```bash
docker build -f Dockerfile -t spacellava-server:latest
docker run -it --rm --gpus all -p8000:8000 -p8001:8001 -p8002:8002 --shm-size 24G spacellama3.1-server:latest
python3 client.py --image_path "https://remyx.ai/assets/spatialvlm/warehouse_rgb.jpg" \
--prompt "What is the distance between the man in the red hat and the pallet of boxes?"
```
## Citation
```
@article{chen2024spatialvlm,
title = {SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities},
author = {Chen, Boyuan and Xu, Zhuo and Kirmani, Sean and Ichter, Brian and Driess, Danny and Florence, Pete and Sadigh, Dorsa and Guibas, Leonidas and Xia, Fei},
journal = {arXiv preprint arXiv:2401.12168},
year = {2024},
url = {https://arxiv.org/abs/2401.12168},
}
@inproceedings{karamcheti2024prismatic,
title = {Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models},
author = {Siddharth Karamcheti and Suraj Nair and Ashwin Balakrishna and Percy Liang and Thomas Kollar and Dorsa Sadigh},
booktitle = {International Conference on Machine Learning (ICML)},
year = {2024},
}
``` |