SAMTAlign A MultiStage Transformer for InstructionBased Captioning and Editing

This project presents a Django-based web application implementing a multi-stage transformer pipeline for instruction-based image captioning and semantic editing. The system integrates state-of-the-art AI models to provide an intuitive user interface for automated image understanding and manipulation. Upon image upload, the BLIP (Bootstrapping Language-Image Pre-training) model generates descriptive captions, which are stored in user history. For editing, the application employs GroundingDINO for precise object localization based on source and target text prompts. The Segment Anything Model (SAM) generates high-fidelity masks for identified objects. Finally, a Stable Diffusion Inpainting pipeline performs context-aware image modifications, seamlessly replacing, removing, or altering objects as per user instructions. The Django framework manages user registration, session handling, and persistent storage of original and edited images. This fusion of large language models and diffusion techniques within a web framework demonstrates a powerful, accessible tool for interactive and automated image content creation.

Keywords: Django, BLIP, Image Captioning, GroundingDINO, SAM, Stable Diffusion, Inpainting, Transformers, Image Editing, Deep Learning, Computer Vision