Scaling Web Scraping with Proxies

Proxies are game-changers in web scraping, allowing you to bypass rate limits, dodge anti-bot measures, and operate with a little more stealth. If you’re working with Python, then the Requests library is your best friend for HTTP connections. It's simple, powerful, and easy to integrate with proxy services.
But here's the thing—using proxies with Requests isn’t always as straightforward as it seems. There are a few pitfalls you can avoid, and some strategies you should implement from the get-go to make your scraping as efficient as possible.
Let’s break it down, step-by-step, and explore how you can integrate proxies into your Python Requests workflow.

Using Proxies with Python Requests

First, let's dive into the basics. Proxies are essential in many scraping workflows, whether you're crawling through websites at scale or simply trying to avoid hitting rate limits.
To start, you’ll need to install the requests module if you haven’t already:

python3 -m pip install requests

Now, let’s get to the good stuff. Here's a simple example of how to use a proxy with the requests library:

import requests

http_proxy = "http://130.61.171.71:3128"
proxies = {
    "http": http_proxy,
    "https": http_proxy,
}

response = requests.get("https://ifconfig.me/ip", proxies=proxies)
print(response, response.text)

When you run this, you'll get the IP address of the proxy you’re using. It’s that simple.
If you’re using free proxies, be aware that they can be unreliable and may not work when you try them.

Proxies’ Dictionary Syntax – Why It Matters

The magic happens in the proxies dictionary. It maps a protocol (like HTTP or HTTPS) to the proxy’s URL. But why are there two http and https entries? Why not just one? Here’s the key:

proxies = {
  "http": "http://proxy_host:proxy_port",
  "https": "https://proxy_host:proxy_port"
}

The reason for using both is that different protocols need separate handling. If you're scraping secure sites (which you probably are), make sure you include https. Not doing so could result in a nasty surprise: SSL errors.

What You Should Know About Proxy Connection Types

Proxies come in various flavors. Let’s break them down.

1. HTTP Proxies: The standard. Fast and reliable, but the traffic isn’t encrypted. Use it for less sensitive tasks.
2. HTTPS Proxies: Similar to HTTP but with SSL/TLS encryption. Slower but secure.
3. SOCKS5 Proxies: Great for flexibility. Can handle a wide range of protocols, including Tor. If you want DNS resolution to happen on the proxy’s server, go with socks5h.
   To use SOCKS5 with Requests, you'll need an additional library:

python3 -m pip install requests[socks]

Then, configure it like this:

import requests

socks5_proxy = "socks5://username:[email protected]:1080"
proxies = {
    "http": socks5_proxy,
    "https": socks5_proxy,
}

response = requests.get("https://ifconfig.me", proxies=proxies)
print(response, response.text)

Always Use Proxy Authentication

Free proxies are usually unreliable, so don’t even bother. In the real world, you’ll need authenticated proxies. Here’s how to set it up:

username = "your_username"
password = "your_password"

proxies = {
    "http": f"http://{username}:{password}@proxy.example.com:1080",
    "https": f"https://{username}:{password}@proxy.example.com:443"
}

Make sure you’re using a reliable proxy provider. This is key if you're scraping at scale or need consistency.

Simplifying Proxies with Environment Variables

You can set proxy variables directly in your environment, which simplifies your code. Here’s how:

$ export HTTP_PROXY='http://username:[email protected]:1080'
$ export HTTPS_PROXY='https://username:[email protected]:443'

Once set, you can make requests without specifying the proxies in your code:

import requests

response = requests.get("https://ifconfig.me/ip")
print(response.text)

Using Requests Session for Proxy Configurations

A Session object in Requests is like a reusable container where you can set defaults, such as headers, timeouts, and—of course—proxies. Here’s a simple use case:

import requests

session = requests.Session()
session.proxies.update({
    "http": "http://username:[email protected]:1080",
    "https": "https://username:[email protected]:443"
})

response = session.get("https://ifconfig.me/ip")
print(response.text)

This method ensures that the proxy settings persist across multiple requests.

Scaling Scraping with Proxy Rotation

If you're scraping large amounts of data, rotating your proxies is essential to avoid getting blocked. Here’s how you can rotate between a list of proxies:

import random
import requests

proxies_list = [
    "http://proxy1:8080",
    "http://proxy2:8080",
    "http://proxy3:8080"
]

for _ in range(10):
    proxies = {"https": random.choice(proxies_list)}
    response = requests.get("https://ifconfig.me/ip", proxies=proxies)
    print(response.text)

But wait—if you’re using a professional proxy provider, proxy rotation is a breeze. Just configure it once, and you can rotate through millions of proxies seamlessly.

Sticky vs. Rotating Proxies

Some tasks require using the same proxy for multiple requests (sticky proxies), while others need to rotate frequently (rotating proxies). For example, if you’re scraping product prices from different Amazon regions, a sticky proxy will ensure you don’t get flagged.
Here’s an example of how you can set up sticky proxies:

import requests
from uuid import uuid4

def sticky_proxies():
    sessions = [uuid4().hex[:6] for _ in range(2)]
    for i in range(10):
        session = sessions[i % len(sessions)]
        proxy = f"http://username,session_{session}@proxy.example.com:1080"
        proxies = {"http": proxy, "https": proxy}
        response = requests.get("https://ifconfig.me/ip", proxies=proxies)
        print(f"Session {session}: {response.text}")

sticky_proxies()

In this setup, the proxy switches every few requests, but the same session name ensures consistent IP usage within that session.

Common Errors and How to Handle Them

When working with proxies, errors are inevitable. The key is to handle them gracefully. Common issues include:

• Timeouts
• Connection errors
• SSL errors

A simple retry mechanism can help, or you can disable SSL verification when needed:

import requests
import urllib3

# Disable SSL warnings
urllib3.disable_warnings()

response = requests.get("https://ifconfig.me/ip", proxies=proxies, verify=False)
print(response.text)

Wrapping Up

Proxies are essential when scraping the web. They help scale your efforts, protect your IP address, and avoid blocks. However, using them effectively requires understanding the right syntax, knowing when to rotate IPs, and handling errors efficiently.