---
license: llama3.1
datasets:
- remyxai/vqasynth_spacellava
tags:
- remyx
---

![image/png](https://cdn-uploads.huggingface.co/production/uploads/647777304ae93470ffc28913/Z7kEAxSxvpYkKNjBLm6GY.png)

# Model Card for SpaceLlama3.1

**SpaceLlama3.1** uses [llama3.1-8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B) as the llm backbone along with the fused DINOv2+SigLIP features of [prismatic-vlms](https://github.com/TRI-ML/prismatic-vlms).


## Model Details

Uses a full fine-tune on the [spacellava dataset](https://huggingface.co/datasets/remyxai/vqasynth_spacellava) designed with [VQASynth](https://github.com/remyxai/VQASynth/tree/main) to enhance spatial reasoning as in [SpatialVLM](https://spatial-vlm.github.io/).

### Model Description

This model uses data synthesis techniques and publically available models to reproduce the work described in SpatialVLM to enhance the spatial reasoning of multimodal models.
With a pipeline of expert models, we can infer spatial relationships between objects in a scene to create VQA dataset for spatial reasoning.


- **Developed by:** remyx.ai
- **Model type:** MultiModal Model, Vision Language Model, Prismatic-vlms, Llama 3.1
- **Finetuned from model:** Llama 3.1

### Model Sources
- **Dataset:** [SpaceLLaVA](https://huggingface.co/datasets/remyxai/vqasynth_spacellava)
- **Repository:** [VQASynth](https://github.com/remyxai/VQASynth/tree/main)
- **Paper:** [SpatialVLM](https://arxiv.org/abs/2401.12168)

## Usage

Try the `run_inference.py` script to run a quick test:
```bash
python run_inference.py --model_location remyxai/SpaceLlama3.1
                        --image_source "https://remyx.ai/assets/spatialvlm/warehouse_rgb.jpg"
                        --user_prompt "What is the distance between the man in the red hat and the pallet of boxes?"
                        
```

## Deploy
Under the `docker` directory, you'll find a dockerized Triton Server for this model. Run the following:

```bash
docker build -f Dockerfile -t spacellava-server:latest
docker run -it --rm --gpus all -p8000:8000 -p8001:8001 -p8002:8002 --shm-size 24G spacellama3.1-server:latest
python3 client.py --image_path "https://remyx.ai/assets/spatialvlm/warehouse_rgb.jpg" \
                  --prompt "What is the distance between the man in the red hat and the pallet of boxes?"
```

## Citation
```
@article{chen2024spatialvlm,
  title = {SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities},
  author = {Chen, Boyuan and Xu, Zhuo and Kirmani, Sean and Ichter, Brian and Driess, Danny and Florence, Pete and Sadigh, Dorsa and Guibas, Leonidas and Xia, Fei},
  journal = {arXiv preprint arXiv:2401.12168},
  year = {2024},
  url = {https://arxiv.org/abs/2401.12168},
}

@inproceedings{karamcheti2024prismatic,
  title = {Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models},
  author = {Siddharth Karamcheti and Suraj Nair and Ashwin Balakrishna and Percy Liang and Thomas Kollar and Dorsa Sadigh},
  booktitle = {International Conference on Machine Learning (ICML)},
  year = {2024},
}
```