File size: 5,596 Bytes
d483e35 d334b38 d483e35 0f239d1 b92bc50 d483e35 c70ed85 d483e35 b29cf22 b92bc50 d334b38 cad1498 d334b38 b29cf22 cad1498 b92bc50 20c8393 3eb8cc5 b92bc50 20c8393 d334b38 20c8393 c70ed85 6aa315d d334b38 32bb3de d334b38 b92bc50 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 |
---
license: apache-2.0
language:
- zh
- en
---
# Pathology-LLaVA-(PCaption-0.5M dataset)
We developed a domain-speciffc large language-vision assistant (PA-LLaVA) for pathology image understanding. Specifically, (1) we first construct a human pathology image-text dataset by cleaning the public medical image-text data for domainspecific alignment; (2) Using the proposed image-text data, we first train a pathology language-image pretraining (PLIP) model as the specialized visual encoder for pathology image, and then we developed scale-invariant connector to avoid the information loss caused by image scaling; (3) We adopt two-stage learning to train PA-LLaVA, first stage for domain alignment, and second stage for end to end visual question & answering (VQA) task.
##### Our code is publicly available on Github.[ddw2AIGROUP2CQUPT/PA-LLaVA (github.com)](https://github.com/ddw2AIGROUP2CQUPT/PA-LLaVA)
## Architecture
## Human Pathology Image-Text data （PCaption-0.5M）
### Introduction
These public datasets contain substantial amounts of data unrelated to human pathology. To obtain the human pathology image-text data, we performed two cleaning processes on the raw data, as illustrated in the follow figture: (1) Removing nonpathological images. (2) Removing nonhuman pathology data. Additionally, we excluded image-text pairs with textual descriptions of fewer than 20 words. Ultimately, we obtained 518,413 image-text pairs (named "PCaption-0.5M" ) for the aligned training dataset.
Instruction fine-tuning phase we only cleaned PMC-VQA in the same way and obtained 15,788 question-answer pairs related to human pathology. Lastly, we combined PathVQA and Human pathology data obtained from PMC-VQA, thereby constructing a dataset of 35543 question-answer pairs data.
#### Data Cleaning Process
## Get the Dataset
### Step 1 Download the public datasets.
Here we only provide the download link for the public dataset and expose the image id index of our cleaned dataset on HuggingFace.
#### Domain Alignment Stage
PubMedVision-Alignment: [FreedomIntelligence/PubMedVision · Datasets at Hugging Face](https://huggingface.co/datasets/FreedomIntelligence/PubMedVision)
PMC-OA: [axiong/pmc_oa · Datasets at Hugging Face](https://huggingface.co/datasets/axiong/pmc_oa)
Quilt-1M: [Quilt-1M: One Million Image-Text Pairs for Histopathology (zenodo.org)](https://zenodo.org/records/8239942)
#### Instruction Tuning Stage
PathVQA: https://drive.google.com/drive/folders/1G2C2_FUCyYQKCkSeCRRiTTsLDvOAjFj5
PMC-VQA: [xmcmic/PMC-VQA · Datasets at Hugging Face](https://huggingface.co/datasets/xmcmic/PMC-VQA)
#### Categorical dataset for zero-sample testing
ICIAR 2018 BACH: https://iciar2018-challenge.grand-challenge.org/Download/
OSCC: https://data.mendeley.com/datasets/ftmp4cvtmb/1
ColonPath : https://medfm2023.grand-challenge.org/datasets
### Step 2 Data processing.
First, use the image index of the clean dataset provided by us to extract the human pathological dataset, and then process it into the following format:
```
[
{
"image": ,
"caption":
},
]
```
Finally, run dataformate.py to get the format needed to train the model.
```
python dataformat.py
```
## Checkpoint
Our released weights are distributed training weights that can be directly loaded for training through XTuner. If you need merged weights, they can be merged using XTuner (using the weights from the domain alignment phase as an example):
```
xtuner convert pth_to_hf path/pallava_domain_alignment.py ./domain_alignment_weight.pth ./domain_alignment_weight_ft
xtuner convert merge meta-llama/Meta-Llama-3-8B-Instruct ./domain_alignment_weight_ft/llm_adapter ./domain_alignment_weight_ft/llm_merge_lora
```
## Training
We used xtuner as a training tool, so please go to xtuner official to complete the environment configuration [https://github.com/InternLM/xtuner]. Then place the pallava folder under the xtuner_add folder into the xtuner folder.
#### Domain Alignment
```
NPROC_PER_NODE=8 NNODES=2 PORT=12345 ADDR= NODE_RANK=0 xtuner train pallava_domain_alignment.py --deepspeed deepspeed_zero2 --seed 1024
```
#### Instruction Tuning
```
NPROC_PER_NODE=8 NNODES=2 PORT=12345 ADDR= NODE_RANK=0 xtuner train pallava_instruction_tuning.py --deepspeed deepspeed_zero2 --seed 1024
```
## Result
## Citation
```
@INPROCEEDINGS{10821785,
author={Dai, Dawei and Zhang, Yuanhui and Xu, Long and Yang, Qianlan and Shen, Xiaojing and Xia, Shuyin and Wang, Guoyin},
booktitle={2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)},
title={PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understanding},
year={2024},
volume={},
number={},
pages={3138-3143},
keywords={Connectors;Pathology;Visualization;Codes;Computational modeling;Biological system modeling;Data models;Cleaning;Bioinformatics;Biomedical imaging;Pathology Image Understanding;VQA;LLaVA},
doi={10.1109/BIBM62325.2024.10821785}}
```
## contact
mailto: [S230233056@stu.cqupt.edu.cn](mailto:S230233056@stu.cqupt.edu.cn) or [dw_dai@163.com](mailto:dw_dai@163.com)
|