Getting Started with PSALM
This document provides a brief introduction of the usage of PSALM.
For training and evaluation with PSALM, make sure you have prepared the corresponding dataset as required.
Training in Command Line
PSALM conducts a two stage training strategy.
First stage training following llava, the pretrained projector can be downloaded here
Second stage training:
- Run
bash scripts/train.shto train PSALM. - Note: change model paths and dataset paths to the exact paths in
train.sh
Evaluation in Command Line
(Optional) Download our trained model from here
In-Domain Tasks
- Panoptic COCO
python psalm/eval/panoptic_segmentation.py --image_folder /path/to/coco/val2017/ --model_path /path/to/PSALM --json_path /path/to/coco
- Instance COCO
python psalm/eval/instance_segmentation.py --image_folder /path/to/coco/val2017/ --model_path /path/to/PSALM --json_path /path/to/coco/instance_val_psalm.json
- RefCOCO
python psalm/eval/referring_segmentation.py --image_folder /path/to/train2014 --json_path /path/to/refcoco/refcoco_val.json --model_path /path/to/PSALM
# also can eval refcoco+ and refcocog by replacing --json_path
- Interactive COCO
python psalm/eval/region_segmentation.py --image_folder /path/to/coco/val2017 --model_path /path/to/PSALM/ --json_path /path/to/coco_interactive_val_psalm.json --region_mask_type point_visual_prompt_mask
# also can eval box/scribble/mask by replacing --region_mask_type to box/scribble/mask_visual_prompt_mask
Out-Domain Tasks
- OV segmentation
python psalm/eval/semantic_segmentation.py --model_path /path/to/PSALM --ov_task_list 'pc_20||ctx_459||ctx_59'
# you need to change the dataset paths in semantic_segmentation.py "OV_SEM_DICT"
- gRefCOCO
python psalm/eval/eval_grefcoco.py --model_path /path/to/PSALM --json_path /path/to/grefcoco/grefcoco_val.json --image_folder /path/to/train2014/
- DAVIS
python psalm/eval/eval_davis.py --image_folder /path/to/DAVIS --model_path /path/to/PSALM --json_path /path/to/DAVIS/2017/trainval_val_psalm.json