Self-Hosted Rotating Proxy Network

Collecting search data at scale means sending huge volumes of requests to search engines — which quickly leads to rate limiting and IP bans when traffic comes from a small, static set of addresses. Relying on third-party proxy providers is expensive at volume, offers limited control over regions and rotation, and adds an external dependency to a core part of the business. Nightwatch needed its own private proxy network: one it fully controlled, that could route requests through many fresh IP addresses across different regions and rotate them continuously to stay ahead of blocking.

We built a self-hosted rotating proxy network in Go. It automatically provisions a configurable number of compute instances across multiple AWS regions, connects each one back to a central proxy server over a secure SSH tunnel, and routes incoming requests through a randomly selected instance — so outbound traffic is spread across many IP addresses in different locations. Instances are rotated on a continuous schedule: new ones are brought up and old ones retired on a timer, with in-flight requests allowed to drain before an instance is shut down, so rotation never drops live traffic. The whole network is driven by simple configuration — instances per region, regions, and rotation interval — making it easy to scale capacity up or down.

Nightwatch gained its own private, controllable proxy network instead of depending on external providers, giving it fresh IPs across multiple regions and continuous rotation to keep request traffic flowing without being blocked. Because it's self-hosted on standard cloud compute, capacity and geographic spread are a matter of configuration, and traffic is spread across many rotating addresses by design. Rotation is graceful, draining active requests so reliability isn't sacrificed for freshness.

• Self-Hosted & Controlled: A private proxy network Nightwatch owns end to end, with no third-party proxy dependency.
• Multi-Region IPs: Provisions instances across multiple AWS regions for geographic spread.
• Continuous Rotation: Instances are cycled on a timer to keep IP addresses fresh and avoid bans.
• Graceful Drain: In-flight requests finish before an instance is retired, so rotation never drops traffic.
• Config-Driven Scaling: Instances-per-region and rotation interval are set by simple configuration.
• Secure Tunnelling: Each instance connects back over an SSH tunnel.

• Go: The proxy server, instance management and request routing.
• AWS EC2: On-demand, multi-region compute that provides the rotating IP addresses.
• SSH tunnelling (multiplexed): Secure connections between the proxy server and each instance.
• AWS S3 & IAM: Distribution of the proxy binary to instances under a scoped role.

We started from the core need — a large, rotating pool of IPs under our own control — and built the network around standard cloud compute so it could scale simply and cheaply. The proxy server provisions instances across the configured AWS regions, connects to each over a multiplexed SSH tunnel, and routes each request through a randomly chosen instance to spread traffic across addresses. We made rotation continuous but safe, cycling instances on a timer while draining active requests before shutdown so live traffic is never interrupted. The result is a self-contained, configuration-driven proxy network that Nightwatch operates itself.