High-Volume Product Enrichment Pipeline

Challenge

AI-driven product discovery only works if every product in the catalogue has been processed and kept up to date for search. With a catalogue that constantly changes — new products arriving, old ones dropping out across multiple data sources — doing that processing by hand is impossible. Shareight needed a service that could keep hundreds of thousands of products continuously search-ready, without redoing work already done, and without falling behind as the catalogue grew.

Solution

We built a dedicated vectorization service in Go that continuously keeps Shareight's product catalogue search-ready. It detects which products still need AI processing, generates the data they need using Google Gemini, and makes it available to the search platform — running continuously and at high throughput so the catalogue never falls behind. The service is built to run unattended: it picks up only the work that's outstanding, recovers cleanly from interruptions, and supports a separate test environment for safe validation before changes go live.

Project Results

The pipeline keeps the product catalogue continuously search-ready, powering AI discovery and curation across the platform. Because it only does outstanding work and recovers cleanly from interruptions, it runs unattended and never falls behind.

Features and Benefits

• Continuous Processing: Keeps new and changed products search-ready without manual intervention.
• High Throughput: Processes large catalogues quickly while staying within service limits.
• Safe to Restart: Only does outstanding work, so restarts waste nothing and recover cleanly.
• Test Environment: Supports a separate environment for safe validation before going live.
• Reliable Operation: Built to run unattended with clean handling of interruptions.

Technologies Used

• Go: High-performance, concurrent service for processing large datasets.
• Google Gemini: AI model behind meaning-based product matching.
• Elasticsearch: Source and destination for the search-ready product data.
• PostgreSQL: Catalogue metadata and references.

Development Process

We started from the throughput problem: how to keep a large, constantly-changing catalogue continuously search-ready without re-doing work or overloading external services. The service was designed to stream through unlimited products and to scale its processing to the capacity available. We made it safe to restart from day one so it could run continuously and survive interruptions. Finally, we added a separate test path and full monitoring so the pipeline could be operated reliably in production.