Using a VPS for web scraping, without becoming the problem

Web scraping from a VPS is a legitimate and common use case — but the difference between a competent scraper and an abusive one is entirely in how you behave. This guide covers the setup and, just as importantly, the etiquette and legality.

Why scrape from a VPS at all

• ▹Your home IP is not exposed to the sites you crawl.
• ▹Your connection does not drop when your ISP decides to.
• ▹You can run scheduled jobs continuously without a desktop awake.
• ▹You can rotate egress addresses for large-scale jobs — legitimately, and within rate limits.

The basic setup

A $6–15 VPS is plenty for most scraping. Install Python with requests and BeautifulSoup, or Node with a fetching library, and schedule with cron or a systemd timer. Respect crawl rate with explicit delays — this is the part that separates a good scraper from a bad one.

• ▹Identify your scraper honestly in the User-Agent, with a contact.
• ▹Set a per-domain delay — one request every few seconds is polite for most sites.
• ▹Honour robots.txt unless you have a specific, defensible reason not to, and document it.
• ▹Cache responses locally so you do not re-fetch what you already have.

Note · Crawling a site that forbids it in robots.txt, at volume, ignoring rate limits, is legally risky in several jurisdictions and will get your IP blocked. The technical setup is the easy part; the behaviour is what matters.

Proxies, and when to use them

For large jobs that legitimately need more reach, route some requests through proxies or a second region. Use them to spread load and handle sites with geo-specific content — not to defeat a block someone deliberately put up.

Note that if your goal is specifically anonymity around scraping, paying in Monero for the server and keeping no logs on it does not change the fact that the target site's servers see your IP. That is unavoidable and it is the site you are contacting, not the payment rail, that is the relevant party.

Legal basics

Publicly accessible data is not automatically free to harvest at any rate. Terms of service, robots.txt, database rights and rate limits all carry legal weight in different jurisdictions, and GDPR-style rules apply to personal data regardless of where you are. If a project is commercial or touches personal data, get proper advice rather than relying on this guide.

Operational hygiene

• ▹Keep your scraper's IP clean — do not crawl from an address that other projects depend on.
• ▹Rotate user agents honestly rather than impersonating browsers deceptively.
• ▹Store what you collect responsibly; scraped personal data is still personal data.
• ▹Have a plan for when a site blocks you: back off, do not escalate.

Which plan

Wisp at $6 for small, polite jobs. Pulse or Umbra if you are running many concurrent requests or heavy parsing. You rarely need bandwidth limits — the discipline is in the delay, not the plan size.

FAQ