Technical Musings

Friday, March 3, 2017

How To Avoid Taking Down Your S3 With Ansible

The s3 outage summary ( describing the cause of the outage 2/28/2017, specifically says an 'established playbook' was used to take down s3, which sounds like they used an Ansible playbook.  And I'm pretty sure how such a terrible error happened, because I've worked around this problem with Ansible in my own projects.

The normal way you run a subset of a group of servers in Ansible is to add the '--limit' parameter, which filters the list of servers based on a group name.  So, of example, you say run on 'web' group with a filter of 'webserver01''; this would only run the Ansible playbook on 'webserver01', not on all the 'web' servers.

The problem is, if you badly specify '--limit' or leave it off, it runs against the whole group.  This is a horrible design flaw.

The work around is not to use '--limit' at all, but instead you specify `hosts: "{{ target }}"` in your playbook.  So you must specify '-e "target=webserver01' or you get an error saying no hosts specified.  The target can be a pattern, so "target=web:database" or "target=webserver0*" works, so this is flexible enough to not need '--limit' at all, and avoid this dangerous design flaw of Ansible.

Tuesday, January 6, 2015

'Compressing' CloudFormation template to get around size limit

Recently I hit the 51,200 byte body size limit of AWS's CloudFormation's templates.  I looked into creating Nested Stacks, but that seemed like a pain.  Looking at the created json template, I saw a lot of unneeded whitespace.

I used troposhere to generate the template, so it was easy to reduce the size by stripping out the beginning and ending whitespace of each line in the json file.

I just added the following line:
json_compressed="\n".join([line.strip() for line in t.to_json().split("\n")])

This more than havled the size of the template from ~60K to ~25K bytes:

$ wc template.json
    1821    2860   60133 template.json

$ wc template_compressed.json
    1821    2860   24616 template_compressed.json

And CloudFormation accepted it, no problem.

Friday, January 2, 2015

Elixir Tgraph

In my attempt to learn Elixir, I've converted an Erlang version of a Python script I wrote years ago.  It takes a pipe of numeric values and plots lines in a character terminal.  Super simple, but handy sometimes.

Now, I know line count is a horrible way to compare code, but here it is:

136 tgraph.escript
106 tgraph.ex

The python code is many years old.  Maybe I've learned something since then, but I don't write very compactly, on purpose.  I only use standard libraries since I don't have the luxury of installing stuff on some of the systems I want to run this on, so can't use some of the cool libraries out there.

It's hard to compare the python code to the Erlang/Elixir.

The Erlang code has added cruft on top to run as Escript (so I don't have to compile changes).   I've yet to figure how to enable piping of stdin to Elixir without running a build ('mix').  So I get to compile my interpreted code!

Otherwise, the Elixir code is easier to read and more concise than Erlang, which is no suprise.  Elixir +1.

Friday, November 21, 2014

Number of New Connection Per Second

Under pressure, debuging a production system, given the following question:

"How many new connections per second are we creating to S3?".

My one-liner (Linux):

while true; do diff   <(netstat -an | grep ESTAB | grep ":443 "| grep -v "N.N.N.N:443 " | sort) <(sleep 1; netstat -an | grep ESTAB | grep ":443 "| grep -v "N.N.N.N:443 " | sort) | grep "<" | wc -l;sleep 1;done

Where N.N.N.N is the local IP.  Many better ways to do this, but I had this in a few minutes.


Thursday, August 22, 2013

linux mint privacy/security setup


init.d script
apt-get install sysv-rc-conf
sysv-rc-conf dnsycrypt-proxy on
apt-get install unbound
  name: "."
airplane init.d # cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)

# OpenDNS Fallback (configured by Linux Mint in /etc/resolvconf/resolv.conf.d/tail).

disable dnsmasq (not caching):
airplane init.d # cat /etc/NetworkManager/NetworkManager.conf

disabled dnssec in unbound:
    # DNSSEC validation using the root trust anchor.
#    auto-trust-anchor-file: "/var/lib/unbound/root.key"


non-exit relay:
    bandwidth limit

sudo apt-get install secure-delete

password management:

filesystem encryption

adblock plus (disable reasonable ads)

linux mint 15 (cinnamon) on chromebook pixel

linux mint 15 (cinnamon) on chromebook pixel
(Sorry for the info dump, hopefully will fix soon)

setup mint 15 on chromebook

date/time in panel:

sudo apt-get install gnome-tweak-tool

sudo sh ./

sudo add-apt-repository ppa:zedtux/naturalscrolling
sudo apt-get update

sudo apt-get install naturalscrolling
    enable for trackpad, usb mouse
    enable start on login

    default full zoom
    adblock plus

    disable left tap
    enable two finder scroll
    small amount of acceleration

windows tiling management:
gTile extension:

    add brightness applet to panel

    keyboard shorcuts:
        sudo apt-get install xbacklight
        /usr/bin/xbacklight -dec 10
                        -inc 10

suspend fix:

    tpm_tis force=1 interrupts=0

    disable right lower corner as right click:
     31 ##Section "InputClass"
 32 ##        Identifier "Default clickpad buttons"
 33 ##        MatchDriver "synaptics"

 34 ##        Option "SoftButtonAreas" "50% 0 82% 0 0 0 0 0"
 35 ##EndSection

 sudo apt-get install oxygen-cursor-theme
 settings...theme..other settings..mouse pointer: oxy-white

    arrow over to Tunables
    hit enter on each 'bad', turn it to 'good'
    esc to exit

    change whether plugged in/not
        add output of "sudo powertop --csv=powertop.csv" to battery /etc/pm/power.d/power

disable bluetooth by default:
    Run gksu gedit /etc/rc.local and add this before line with exit 0:
    rfkill block bluetooth


Cinnamon desktop optimizations:

windows management:

keyboard launcher:

window borders: HighContrast
check "show icons on buttons"

fix comixcursors
apt-get install comixcursors
get .tar.bz2
extract to:
reload cinamon alt-f2 "r"

tar xjf ComixCursors-0.7.3.tar.bz2

Wednesday, June 5, 2013

monkeyrunner hanging on input/raw_input on Mac OS X: RESOLVED

I found recently that my monkeyrunner scripts started failing after updating the Android SDK Tools to version 22.0.1.  This is a known bug in the jython version shipped with it.  Luckily it's easily fixed by replacing one .jar file.

Jython 2.5.4rc1 Release Notes
Bugs Fixed:

[ 1972 ] jython 2.5.3 sys.stdin.readline() hangs when jython launched as subprocess on Mac OS X

1)  Download latest jython 2.5.4rc1:

2) Copy into ${ANDROID_ROOT}/sdk/tools/lib/

3) move or delete the existing jython-standalone-2.5.3.jar

That's it!  monkeyrunner now process raw_input() and input() correctly.