Inhabitr Design Engine 1.0

PaliGemma Vision Model(unifying attributes model)

Leveraging the PaliGemma vision model to unify attribute extraction, this custom trained Model identifies and outputs furniture attributes—such as primary color and secondary color, primary material, secondary material, and design style—in JSON format.

Features

Paligemma Model: A 3-billion parameter multi-model capable of generating captions from images, performing object detection, object segmentation ,OCR and VQA. Custom Training: The model is custom trained on approximately 500 pairs of cropped images (sofa, accent chair) along with their captions.

Requirements

Python 3.9+

Minimum 12GB of GPU for local inference

Pip (Python package installer)

Installation

First, clone the repository:

 git clone git clone https://Apoorva_inhabitr@bitbucket.org/Inhabitr/vision_models.git

Create and Activate Python Virtual Environment

For Unix/macOS

 python3 -m venv env
 source env/bin/activate

For Windows

 python -m venv env
 .\env\Scripts\activate

Install the required Python libraries

pip install -r requirements.txt

Download Model Weights and Tokenizer

Download the model weights and tokenizer from Google Drive

https://drive.google.com/drive/folders/1S-z374V-yd3izeBatAMQlitmZ0mbMv6s?usp=drive_link.

Place the downloaded files in the models/ folder located in the root directory of the project.

Start the API

 python main.py

If running locally, the API will be available at http://localhost:5000.

API Endpoints for Captions

predict attributes

POST /predict

Request Parameters

Send an image file

image: The image file used as the basis for the search.

Example Request with Image File

  curl -X POST http://127.0.0.1:5000/predict \
  -H "Content-Type: multipart/form-data" \
  -F "file=@/path/to/your/image.jpg"

Performance Considerations

GPU: Model parameters are loaded into memory for fast inference within 1 sec after initial loading.

CPU: Initial model loading may take 4-6 minutes based on your system configuration.