Nightwatch — AI Search Visibility Tracking

Search is no longer just blue links. More and more people get answers from AI assistants — ChatGPT, Perplexity, Gemini, Claude and Google's AI overviews — and brands increasingly care whether they're mentioned and cited in those answers, not just where they rank on Google. But this "AI search visibility" is hard to measure: the answers are generated, vary over time, and live across many different AI providers. Nightwatch wanted to give its users a way to track exactly how visible they are across the AI tools their customers actually use.

We built an AI search visibility tracking system that monitors how often a brand or website appears in answers from multiple AI providers. For a given set of prompts, it regularly queries the major AI assistants — including ChatGPT, Perplexity, Gemini and Claude — captures their answers, and extracts the brands and citations mentioned, tracking each one's presence and position over time. Entity extraction combines straightforward citation parsing with AI-based detection so mentions are caught reliably, and a smart refresh schedule keeps results current without wasteful re-querying. The data feeds time-series analytics — visibility share, average position and trends — and a provider registry makes it straightforward to add new AI tools as they emerge. The collection work is distributed and can auto-scale to handle volume.

Nightwatch users can now see their presence in AI-generated answers the same way they track search rankings — across all the major AI assistants, over time. This puts a number on "answer engine optimisation," an area most tools can't measure yet, and ties it back to the broader ranking picture. The modular provider system means coverage can expand as new AI tools appear, and the distributed collection scales with demand.

• Multi-Provider Tracking: Monitors brand presence across ChatGPT, Perplexity, Gemini, Claude and more.
• Mention & Citation Extraction: Reliably detects which brands and sources each AI answer references.
• Visibility Over Time: Time-series analytics for visibility share, average position and trends.
• Smart Refresh: Keeps results current without unnecessary re-querying.
• Extensible by Design: A provider registry makes adding new AI tools straightforward.
• Scalable Collection: Distributed, auto-scaling job processing handles growing volume.

• Python & Go: AI answer collection and parallel job processing.
• OpenAI (and other AI providers): Both as tracked answer engines and for entity extraction.
• PostgreSQL: Time-series storage for mentions, positions and analytics.
• Object storage (S3 / Backblaze B2): Backup of collected answer data.
• Browser automation: Reliable capture of answers from JavaScript-heavy AI interfaces.

We started from the measurement problem: AI answers are generated and inconsistent, so visibility has to be sampled over time across many providers. We built a modular provider system so each AI assistant could be queried through a common interface, and a capture layer robust enough to handle their dynamic interfaces. Extracting *who* is mentioned combined direct citation parsing with AI-based entity detection for reliability. We added a smart refresh schedule to keep data fresh efficiently, stored everything as time-series for trend analytics, and made the collection layer distributed and auto-scaling so it could grow with the number of prompts and providers tracked.