Why Are E-Commerce Searches Failing Shoppers? AI May Have the Answer

Researchers developed a fashion-focused prototype that can respond to text or image searches, write product descriptions, recommend related items, and generate pictures from shoppers’ ideas.

You can picture the shoes you want: white with teal and orange details, finished with a translucent sole. Getting a shopping website to find them can take several searches, especially when the seller's description leaves out the details you care about. A product photograph may contain the answer, but a search engine built around matching words can miss it.

A study titled "A Novel E-Commerce Search Engine Based on Multimodal Generative AI for Improving Product Accessibility," published in the journal AI, explores a search system that uses product photographs and written descriptions together. Abdulrahman Hassan and colleagues developed a fashion-focused prototype that can respond to text or image searches, write product descriptions, recommend related items, and generate pictures from shoppers' ideas.

Their laboratory tests recorded an increase in search accuracy from 70% to 95% after adapting the AI models to shopping data. That figure describes how often the intended product appeared among the first five results, giving readers a more useful picture of what the improvement actually means.

Teaching Search to Notice the Details Shoppers Care About

The researchers assembled more than 250,000 product samples containing images, descriptions, and links, covering men's, women's, and children's clothing alongside accessories such as watches and bags. Sources included public or licensed product databases, the Kaggle Fashion dataset, and a supplementary collection from retail websites.

Preparing those records involved standardising images, removing unsuitable data, checking links, and using GPT-4 to strip unnecessary material from descriptions. LLaVA, an AI model that connects images with language, was then adapted to produce descriptions focused on the products themselves.

The paper's sneaker example shows why that training mattered. The original model could describe a shoe surrounded by colourful balls against a blue background, spending its attention on the setting. The adapted model was trained to focus on information useful for shopping, such as the sneaker's colours and translucent sole.

Training examples paired each image with five questions about the product, encouraging the model to describe specific attributes. The team used an 80/20 training and validation split for LLaVA and the search-matching model, CLIP.

Connecting Pictures, Words, and Product Suggestions

CLIP converts images and text into numerical representations that make their meanings comparable. A shopper's description and a product photograph can receive similar representations even when the listing does not repeat the shopper's exact words.

These representations are stored in ChromaDB, a database designed to retrieve similar items. A text query or uploaded image goes through the same encoding process, allowing the system to compare it with stored products and return relevant matches.

Related-product recommendations use the current query and similarities between items. The paper describes a recommendation approach that does not depend on keeping persistent user histories, although broader privacy protections remain part of the work needed before deployment.

The prototype's interface brings together search, recommendations, image generation, and automatic captioning. Sellers can upload photographs to obtain descriptions that could make their listings easier to discover.

Stable Diffusion supplies the image-generation feature, turning a written idea into a picture of a desired product. A shopper could use that picture to search for something similar or communicate a custom-order concept. Such an image represents an idea rather than proof that the pictured product exists, and the researchers acknowledge that generated pictures can contain invented features.

What Improved in the Tests

The strongest overall result came from comparing the search system before and after adapting its main models. The intended product appeared within the first five results for 95% of test queries, up from 70%, an improvement of 25 percentage points. Reported response time fell from 2.5 seconds to 1.2 seconds.

Forty participants completed search tasks on both versions and rated relevance, efficiency, and their experience. Their combined ratings, converted into percentages, rose from 75% to 95%. These figures describe feedback from a controlled study rather than satisfaction across a live shopping platform.

The description-writing model improved across all reported automated measures of wording and meaning. In a separate assessment, 40 university students evaluated descriptions for 10 images, producing an average score of 80.57 out of 100 for the adapted LLaVA model, compared with 62.89 for the original.

Tests of CLIP offered a useful finding about computing demands. The smaller, adapted model came close to a larger model on matching image–text pairs, with average similarity scores of 31.52% and 33.29%. Incorrect pairs received lower similarity scores from the smaller model, 20.47% compared with 23.18%, suggesting it was better at separating mismatches in that comparison. These are similarity scores, not search-accuracy percentages. The paper describes favourable assessments of generated-image quality and alignment with prompts, but provides less numerical detail for that feature than for search and captioning.

What Shoppers and Smaller Sellers Still Need to Know

Better descriptions and searches based on product attributes could help smaller sellers whose listings are difficult to find. The Researchers present fairer product discovery as a central motivation, but the reported tests do not establish that smaller businesses gained visibility or sales. The study compares versions of its own system; it does not demonstrate that the prototype outperforms major shopping platforms in everyday use.

Its evidence comes primarily from fashion products and English-language data. Performance across other product categories, languages, heavy customer traffic, or confusing and deliberately misleading queries needs further testing. The small caption-rating exercise also limits how broadly its human evaluations can be interpreted.

Models trained on internet data can carry gender, racial, and other biases into search results. The paper discusses these concerns without demonstrating that its system has eliminated them. Protecting uploaded images and text will require stronger safeguards as the technology moves towards real-world use.

Proposed developments include conversational shopping assistants, personalised styling, and generating three-dimensional product views from two-dimensional images. The present study offers a more immediate lesson: teaching search systems to connect the details in photographs with the words people use can make a measurable difference to finding the right item.