Technical Musings: Python Requests proxy all protocols through single URL

Thursday, August 1, 2024

Python Requests proxy all protocols through single URL

The Python Requests library requires you to specify multiple proxy URLs per protocol, 

like this:

import requests
proxies = {
    'http': 'http://10.10.1.10:3128',
    'https': 'http://10.10.1.10:1080', }
requests.get('http://example.org', proxies=proxies)
The annoying thing is if you leave off either protocol, 
it does not error - it just doesn't proxy that traffic.
While this flexibility is great, it would be nice to have
both http and https traffic go through the same proxy
without repeating oneself.
 
A somewhat silly way to make this more DRY is to use my favorite collection: 
defaultdict ( and lambda ).

import requests
from collections import defaultdict
proxies = defaultdict(lambda: 'http://10.10.1.10:3128') requests.get('http://example.org', proxies=proxies)
A defaultdict is usually given a default value of a type, like a list or an int.

Using lambda allows you to give it a specific value, in this case a string.

No comments: