Technical Report: AI-Driven Artist Discovery via Semantic Search
The architecture behind ARTBAT's semantic search: CLIP over an LLM, the embedding pipeline, and AWS deployment.
Companion piece to the ARTBAT Live Case Study. This report covers the architecture and the tool-choice reasoning behind it.
The Problem, Technically
ARTBAT LIVE’s discovery experience depended on manual keyword tagging. Artists tagged their own work; clients searched by exact keyword match. This failed in both directions as the community scaled. Tagging is unpaid labor that artists skip or do inconsistently, and lexical search misses relevant work whenever the client’s vocabulary differs from the artist’s. A query like “a brutalist style with two men standing in front of a doorway” returns nothing useful in a keyword system, no matter how well the artwork matches.
The requirement was semantic search: matching on what a query means and what an artwork actually looks like, with no tagging step anywhere in the pipeline.
Why CLIP and Not an LLM
We worked backwards from the outcome (clients matched to artists based on the visual qualities of the work) to the tools that enable it. Artist discovery is a similarity problem, not a generation problem, and that determines the tool choice:
CLIP encodes images and text into the same vector space. A client’s text description and an artist’s actual artwork become directly comparable vectors; similarity is a dot product, computed in milliseconds against the full library.
An LLM rebuilds the bottleneck we were removing. An LLM cannot compare a query against a million images directly; each artwork would first need to be described in words (captions or tags), reintroducing the tagging step with added latency and inference cost. The intermediate text also discards exactly the visual information (style, composition, palette) that clients search by.
Retrieval is cheap and predictable. Every search is an embedding lookup, not a generation call; there’s no sampling variance in results, latency is measured in milliseconds, and costs scale with uploads rather than usage.
The Model
CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on image and text pairs. We fine-tuned it on style labels drawn from ARTBAT’s own community (for example, an artwork paired with the description “Naruto in chibi style”), so the embedding space reflects the styles ARTBAT’s clients actually search for.
An embedding is a mapping from images and words to vectors of real numbers, where distance corresponds to meaning; the word “brutalist” sits closer to a certain architectural style than to the word “vehicle.” Because images and text share one space, search-by-description and search-by-reference-image are the same operation under the hood.
The system works in three stages:
Train CLIP on style-labeled image and text pairs.
Create the image embedding database. Every uploaded artwork is encoded and stored.
Find digital art using text or images. Incoming queries are encoded and ranked against stored embeddings by similarity.
The Artist Workflow
Upload Art. Artists upload a portfolio of images, along with traditional metadata such as artwork titles, profile contact information, and website links.
Create Embedding. The CLIP model generates an embedding for each uploaded image; no manual tagging exists anywhere in this flow.
Store Embedding. The embedding and its metadata are stored as a single object in an AWS OpenSearch Service k-NN index.
The Client Workflow
A client query, whether text or an uploaded image, is encoded through the same model. The query embedding is compared against stored vectors and artworks are returned ranked from most to least similar. Each stored embedding carries metadata identifying its source image, so results resolve back to the artwork and the artist’s profile, contact information, and website; a search result becomes a client connection directly.
Pre- and Post-Filtering
Storing metadata and embedding vectors together in a single OpenSearch object enables filtered vector search:
Pre-filtering applies filters before similarity ranking, effectively ranking a subset of the total vectors; for example, restricting to artists from a certain region before finding the best match.
Post-filtering applies filters after ranking is complete, refining an already-ranked result set.
Features Built on the Embedding Infrastructure
Because every artwork lives in the index as a vector plus metadata, product features are queries rather than new systems:
Portfolio-level search. Similarity ranks can be computed per artwork or aggregated per artist, exposed as a client-side filter for clients who want an artist with a history of similar work.
Up & Coming Artists. A landing-page surface of artwork uploaded in the past thirty days, pulled from the same index.
Deployment
The model and search infrastructure were developed and deployed on AWS, with OpenSearch Service providing the k-NN vector index and filtered query capability. Model efficacy was measured iteratively through real user interactions against the KPIs and North Star metric established at the start of the engagement, so both the interface and the model were evaluated on whether they moved outcomes.
About Rosenblatt
Rosenblatt AI is an AI-native consultancy that builds production AI systems for startups and PE-backed portfolio companies. Our forward-deployed product engineers embed with your team, work backwards from the outcomes you want, and ship. From conversational AI to semantic search, we take products from proof of concept to production.
Have a similar challenge? Click below to schedule a 30-minute meeting with our CEO and founder.