Technical Musings: August 2024

Thursday, August 1, 2024

Python Requests proxy all protocols through single URL

The Python Requests library requires you to specify multiple proxy URLs per protocol, 

like this:

import requests
proxies = {
    'http': 'http://10.10.1.10:3128',
    'https': 'http://10.10.1.10:1080', }
requests.get('http://example.org', proxies=proxies)
The annoying thing is if you leave off either protocol, 
it does not error - it just doesn't proxy that traffic.
While this flexibility is great, it would be nice to have
both http and https traffic go through the same proxy
without repeating oneself.
 
A somewhat silly way to make this more DRY is to use my favorite collection: 
defaultdict ( and lambda ).

import requests
from collections import defaultdict
proxies = defaultdict(lambda: 'http://10.10.1.10:3128') requests.get('http://example.org', proxies=proxies)
A defaultdict is usually given a default value of a type, like a list or an int.

Using lambda allows you to give it a specific value, in this case a string.