The Concurrency Problem
Running a single browser instance is easy. Hell, you can do it in 10 lines of code. But orchestrating a thousand browsers, each scraping different sites, handling failures gracefully, and not getting banned? That's where most teams hit a wall.
We've built scraping infrastructure that processes over 1 million pages per day for clients in real estate, e-commerce, and finance. This isn't theory.it's battle-tested architecture that's been refined through countless production incidents.
Why Playwright Over Puppeteer or Selenium?
Let me save you some time: Playwright is the best tool for modern web scraping. Here's why:
- Multi-browser support - Chromium, Firefox, and WebKit from one API
- Auto-wait - It waits for elements to be ready before interacting
- Network interception - Block ads, images, and tracking scripts to speed up scraping
- Better debugging - Screenshots, videos, and traces make troubleshooting easier
- Stealth mode - Harder to detect than Puppeteer out of the box
Puppeteer is fine for simple tasks, but Playwright's API is just better designed. And Selenium? Come on, it's 2025.
The Architecture: From Prototype to Production
Most scraping projects start the same way: a single script that runs on your laptop. That's fine for testing, but it doesn't scale. Here's how we structure production scraping systems:
Layer 1: Job Queue
We use Redis with Bull for job queuing. Each scraping task is a job: URL to scrape, selectors to use, data to extract. The queue handles retries, rate limiting, and prioritization.
const queue = new Queue('scraping-jobs', {
redis: { host: 'localhost', port: 6379 },
limiter: {
max: 100, // 100 jobs
duration: 60000 // per minute
}
});
queue.process(10, async (job) => {
const { url, selectors } = job.data;
return await scrapePage(url, selectors);
});
Layer 2: Browser Pool
Instead of launching a new browser for each job, we maintain a pool of warm browsers. This cuts startup time from 2-3 seconds to near-zero.
class BrowserPool {
constructor(size = 10) {
this.browsers = [];
this.size = size;
}
async init() {
for (let i = 0; i < this.size; i++) {
const browser = await chromium.launch({ headless: true });
this.browsers.push(browser);
}
}
async getBrowser() {
// Round-robin or least-busy selection
return this.browsers[Math.floor(Math.random() * this.browsers.length)];
}
}
Layer 3: Proxy Rotation
Getting banned is the #1 reason scraping projects fail. We rotate through residential proxies and implement smart retry logic.
Don't cheap out on proxies. We use Bright Data for residential IPs. Yes, it's expensive ($500+/month), but it's worth it. Free proxies will get you blocked in minutes.
Layer 4: Data Pipeline
Scraped data goes into a staging database (PostgreSQL), gets validated and cleaned, then moves to the production database. We never write directly to prod from scrapers.
Handling Anti-Bot Measures
Modern websites are smart. They use Cloudflare, PerimeterX, DataDome, and custom fingerprinting. Here's how we bypass them:
1. Stealth Plugins
Use playwright-extra with stealth plugins to mask automation signals:
const { chromium } = require('playwright-extra');
const stealth = require('puppeteer-extra-plugin-stealth')();
chromium.use(stealth);
const browser = await chromium.launch();
2. Realistic User Behavior
Don't just click and extract. Scroll, move the mouse, wait random intervals. Make it look human.
await page.mouse.move(100, 200);
await page.waitForTimeout(Math.random() * 2000 + 1000);
await page.mouse.wheel(0, 100);
3. Rotate User Agents and Fingerprints
Change browser fingerprints for each session. This includes canvas fingerprints, WebGL, and audio context.
4. Solve CAPTCHAs
Yes, we use 2Captcha and Anti-Captcha services. It's not elegant, but it works. Budget $0.001-$0.003 per CAPTCHA solved.
Performance Optimization
Scraping 1M pages means every millisecond counts. Here's how we optimize:
Block Unnecessary Resources
Images, fonts, and analytics scripts slow you down. Block them:
await page.route('**/*', (route) => {
const url = route.request().url();
if (url.includes('google-analytics') || url.endsWith('.png')) {
route.abort();
} else {
route.continue();
}
});
Use Network Interception
Sometimes you don't need to render the page at all. Intercept the API calls and grab data directly:
page.on('response', async (response) => {
if (response.url().includes('/api/products')) {
const data = await response.json();
console.log('Got data directly from API:', data);
}
});
Parallel Processing
Run multiple pages in the same browser context. We typically run 5-10 pages per browser instance.
Error Handling and Monitoring
Scrapers fail. A lot. Sites change their HTML, proxies die, servers timeout. You need robust error handling:
async function scrapePage(url, retries = 3) {
for (let i = 0; i < retries; i++) {
try {
const page = await browser.newPage();
await page.goto(url, { timeout: 30000 });
const data = await extractData(page);
await page.close();
return data;
} catch (error) {
console.error(`Attempt ${i + 1} failed:`, error.message);
if (i === retries - 1) throw error;
await sleep(Math.pow(2, i) * 1000); // Exponential backoff
}
}
}
We use Sentry for error tracking and Grafana for monitoring. Key metrics:
- Success rate (target: >95%)
- Average scrape time (target: <5s per page)
- Proxy health (ban rate, response time)
- Queue depth (how backed up are we?)
Kubernetes Deployment
We run scrapers on Kubernetes for easy scaling. Here's a simplified deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: scraper-workers
spec:
replicas: 10
template:
spec:
containers:
- name: scraper
image: our-scraper:latest
resources:
requests:
memory: "512Mi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "2000m"
Each pod runs 2-3 browser instances. We scale horizontally based on queue depth.
Cost Breakdown
Scraping at scale isn't cheap. Here's what it costs us to scrape 1M pages/day:
- Proxies: $1,500/month (residential IPs)
- Compute: $800/month (Kubernetes cluster)
- Storage: $200/month (PostgreSQL + backups)
- CAPTCHA solving: $300/month (varies by site)
- Total: ~$2,800/month
That's $0.0028 per page. For clients, we charge $0.01-$0.05 per page depending on complexity.
Legal and Ethical Considerations
I'm not a lawyer, but here's what we follow:
- Respect robots.txt (mostly)
- Rate limit to avoid DDoSing sites
- Only scrape public data
- Don't scrape personal information without consent
- Have clients sign agreements acknowledging legal responsibility
The legal landscape is murky. The HiQ vs. LinkedIn case set some precedents, but it's not settled law. Be careful.
Lessons Learned
After building dozens of scrapers, here's what I wish I knew earlier:
1. Sites will change
Build scrapers to be resilient to HTML changes. Use multiple selectors, fallback logic, and automated alerts when extraction fails.
2. Proxies matter more than code
You can have perfect code, but if your proxies are bad, you'll get banned. Invest in quality proxies.
3. Start simple, scale later
Don't over-engineer. Start with a single script, validate the data quality, then scale.
4. Monitor everything
You can't fix what you can't see. Log everything, track metrics, set up alerts.
Want Help?
We build custom scraping infrastructure for clients who need reliable, high-volume data extraction. If you're scraping more than 10K pages/day and hitting scaling issues, let's talk.
