A Hybrid Framework Combining Preprocessing, VGG-19 Transfer Learning, and GPT Enhanced CLIP for Fine-Grained Classification

Abstract

Contrastive vision-language models (VLMs) like CLIP have shown that they can do well on zero-shot classification tasks, but their accuracy relies a lot on how effectively the prompts are built. To solve this problem, we use GPT-4 to make prompts that are more visually descriptive, which makes CLIP more flexible when working with fine-grained picture collections. We put up a whole pipeline on the CUB-200 bird species dataset, starting with preprocessing the images (resizing, converting to greyscale, and bilateral filtering) and then utilizing statistical measures and the Gray-Level Co-occurrence Matrix (GLCM) to extract features. The dataset is divided into training and testing sets, and VGG-19 is used for transfer learning to classify the data. This gives a solid visual baseline. To make the text space more interesting, GPT-4 is used to provide extensive visual explanations for each class label. These descriptions are then added to CLIP’s text embedding space. These embeddings match the outputs of CLIP’s visual encoder, which makes it easier to interpret things from different modes. Using criteria like accuracy, F1-score, precision, recall, and mistake rate to test the suggested technique shows that it works better than typical CLIP prompts for categorization. Finally, a web site is built using Python, Streamlit, HTML, and CSS that lets users input pictures and get predictions with full details. This study shows how feature-based preprocessing, deep transfer learning, and GPT-4-enhanced prompt engineering may work together to greatly improve zero-shot classification accuracy in fine-grained image recognition tasks.