Python Requests proxy all protocols through single URL

The Python Requests library requires you to specify multiple proxy URLs per protocol, 

like this:

import requests
proxies = {
    'http': 'http://10.10.1.10:3128',
    'https': 'http://10.10.1.10:1080', }
requests.get('http://example.org', proxies=proxies)

The annoying thing is if you leave off either protocol, 

it does not error - it just doesn't proxy that traffic.

While this flexibility is great, it would be nice to have

both http and https traffic go through the same proxy

without repeating oneself.

A somewhat silly way to make this more DRY is to use my favorite collection: 

defaultdict ( and lambda ).

import requests
from collections import defaultdict
proxies = defaultdict(lambda: 'http://10.10.1.10:3128') requests.get('http://example.org', proxies=proxies)

A defaultdict is usually given a default value of a type, like a list or an int.

Using lambda allows you to give it a specific value, in this case a string.

I found this template for Elixir scripts to be really useful:

https://arathunku.com/b/2024/shell-scripting-with-elixir/

To add to that, I wanted to require a particular version of Elixir:

elixir_version_required="~> 1.14" if not Version.match?(Version.parse!(System.version), Version.parse_requirement!(elixir_version_required)) do IO.write("Elixir version doesn't match requirement: #{elixir_version_required}") System.halt(0) end

Export Gnome Extension URLs

To export text list of the URLs of your installed Gnome Extensions:

sudo apt-get install jq

cat ~/.local/share/gnome-shell/extensions/*/metadata.json | jq '.url'

This works for most, but some do not include url in metadata.json.

How To Avoid Taking Down Your S3 With Ansible

The s3 outage summary (https://aws.amazon.com/message/41926/) describing the cause of the outage 2/28/2017, specifically says an 'established playbook' was used to take down s3, which sounds like they used an Ansible playbook.  And I'm pretty sure how such a terrible error happened, because I've worked around this problem with Ansible in my own projects.

The normal way you run a subset of a group of servers in Ansible is to add the '--limit' parameter, which filters the list of servers based on a group name.  So, of example, you say run on 'web' group with a filter of 'webserver01''; this would only run the Ansible playbook on 'webserver01', not on all the 'web' servers.

The problem is, if you badly specify '--limit' or leave it off, it runs against the whole group.  This is a horrible design flaw.

The work around is not to use '--limit' at all, but instead you specify `hosts: "{{ target }}"` in your playbook.  So you must specify '-e "target=webserver01' or you get an error saying no hosts specified.  The target can be a pattern, so "target=web:database" or "target=webserver0*" works, so this is flexible enough to not need '--limit' at all, and avoid this dangerous design flaw of Ansible.

'Compressing' CloudFormation template to get around size limit

Recently I hit the 51,200 byte body size limit of AWS's CloudFormation's templates.  I looked into creating Nested Stacks, but that seemed like a pain.  Looking at the created json template, I saw a lot of unneeded whitespace.

I used troposhere to generate the template, so it was easy to reduce the size by stripping out the beginning and ending whitespace of each line in the json file.

I just added the following line:
json_compressed="\n".join([line.strip() for line in t.to_json().split("\n")])
utils.validate_cloudformation_template(json_compressed)

This more than havled the size of the template from ~60K to ~25K bytes:

$ wc template.json
    1821    2860   60133 template.json
$ wc template_compressed.json
    1821    2860   24616 template_compressed.json

And CloudFormation accepted it, no problem.

Elixir Tgraph

In my attempt to learn Elixir, I've converted an Erlang version of a Python script I wrote years ago.  It takes a pipe of numeric values and plots lines in a character terminal.  Super simple, but handy sometimes.

Now, I know line count is a horrible way to compare code, but here it is:

150 tgraph.py https://gist.github.com/dgulino/4750099
136 tgraph.escript https://gist.github.com/dgulino/4750118
106 tgraph.ex https://gist.github.com/dgulino/298516f7977c57199a4a

The python code is many years old.  Maybe I've learned something since then, but I don't write very compactly, on purpose.  I only use standard libraries since I don't have the luxury of installing stuff on some of the systems I want to run this on, so can't use some of the cool libraries out there.

It's hard to compare the python code to the Erlang/Elixir.

The Erlang code has added cruft on top to run as Escript (so I don't have to compile changes).   I've yet to figure how to enable piping of stdin to Elixir without running a build ('mix escript.build').  So I get to compile my interpreted code!

Otherwise, the Elixir code is easier to read and more concise than Erlang, which is no suprise.  Elixir +1.

Number of New Connection Per Second

Under pressure, debuging a production system, given the following question:

"How many new connections per second are we creating to S3?".

My one-liner (Linux):

while true; do diff   <(netstat -an | grep ESTAB | grep ":443 "| grep -v "N.N.N.N:443 " | sort) <(sleep 1; netstat -an | grep ESTAB | grep ":443 "| grep -v "N.N.N.N:443 " | sort) | grep "<" | wc -l;sleep 1;done

Where N.N.N.N is the local IP.  Many better ways to do this, but I had this in a few minutes.

Done.