Technical Musings

SSH LocalCommand / OSX Terminal Colors

2012-02-06T19:47:00.000-08:00

If you're like me, you work in many different computing environments: home, test, QA, and production. And it's REALLY important not to get them mixed up. I've always used a color scheme in my ssh/console windows to help me differentiate each. I've manually set the color of each console before I log in, and try not to reuse a console for different environments. It's kind of a pain, and if I'm brain dead enough to not notice what host I'm logged into, chances are high that I'm not going to set the colors correctly either. So I created an OSX (I'm on 10.7) AppleScript to automatically change the Terminal colors for each of my windows/tabs to match the environment that console is logged into.

To trigger the AppleScript, I added it to an until-now-unknown to me ssh client setting: LocalCommand. It will run a command locally each time you make an ssh connection.

To setup the trigger, you must add the following lines into ~/.ssh/config:
(Note: it's important to add the & at the end of LocalCommand to put the command in background so it can wait for Terminal to connect)

 Host *
  
   PermitLocalCommand yes
  
   LocalCommand /Users/MYNAME/applescript/term_tab_style.scpt &

UPDATE:And this in /etc/ssh_config:

  PermitLocalCommand yes

The AppleScript goes through all open Terminal windows/tabs and checks the title. This is normally set to the user@hostname, and I have a naming convention 'purpose'-'env'-'location'-'number'. Applescript has very limited parsing capabilites; I found the functions I used here to split the hostname up and get the environment name. You can then set the Terminal window/tab to whatever Profile you desire.

The code for term_tab_style.scpt. Copy this code into wherever you specified the LocalCommand in ~/.ssh/config and don't forget to run 'chmod u+x ~/term_tab_style.scpt'.

 #!/usr/bin/osascript
  
 on run argv

   delay 1
  
   tell application "Terminal"
     repeat with w from 1 to count windows
       repeat with t from 1 to count tabs of window w
         set title to custom title of tab t of window w  
         try
           set myArray to my theSplit(title, "@")
           set userName to my getArrayValue(myArray, 1)
           set restName to my getArrayValue(myArray, 2)
           set myArray to my theSplit(restName, "-")
           set envName to my getArrayValue(myArray, 2)
           if envName is "pro" then
             #display dialog ("" & envName)
                set current settings of tab t of window w to (first settings set whose name is "Basic Blue")
           else if envName is "qa" then
                set current settings of tab t of window w to (first settings set whose name is "Basic")
           end if
         end try
       end repeat
     end repeat
   end tell
  
 end run
  
 on theSplit(theString, theDelimiter) 
   -- save delimiters to restore old settings
   set oldDelimiters to AppleScript's text item delimiters
   -- set delimiters to delimiter to be used
   set AppleScript's text item delimiters to theDelimiter
   -- create the array
   set theArray to every text item of theString  
   -- restore the old setting  
   set AppleScript's text item delimiters to oldDelimiters  
   -- return the result  
   return theArray 
 end theSplit
  
 on getArrayValue(array, location)  
   -- very important -- The list index starts at 1 not 0  
   return item location in array  
 end getArrayValue

Python JSON one-liner

2012-01-12T12:02:00.000-08:00

I've seen the python one-liner to parse and pretty-print json:

cat test.json | python -mjson.tool

But I still have to grep/awk to get the value. Here's a python one-liner to get a json value:

curl -s -u user:password "http://test.json.output"  | python -c "import json,sys;input=json.load(sys.stdin);print (input.get('messages'))"

A reasonably short, pure python way to do this is with the requests library, but that's not built into python, and I find one-liners are only useful if they work across a lot of systems as-is:

import json,requests
r = requests.get("http://test.json.output/api/", auth=('user','pass'))
o = json.loads(r.content)
print (o.get('messages'))

Wireshark on OS X

2011-12-06T19:13:00.001-08:00

I run Wireshark (formerly Ethereal) on OSX, but by default only root has rights to the ethernet devices. So either you run it as root, which is a security risk, or you give your own user the rights. The devices (/dev/bf*) that are used are recreated every boot, so just 'chown'ing them won't do.

First add your user to the wheel group:

dscl . append /Groups/wheel GroupMembership 'username'

The run this command:

sudo chmod g+rw /dev/bpf*; open /Applications/Wireshark.app

I added that command to my .profile file as an alias for convenience:

alias wireshark='sudo chmod g+rw /dev/bpf*; open /Applications/Wireshark.app'

MSudo: Mac OSX Graphical SUDO

2011-12-06T18:47:00.001-08:00

I develop a lot on OSX and there are times I need to run a GUI app as root, like a gui editor of a systems file, or running Wireshark. I could just run it from a terminal using "sudo", but that's not very cool. There is the Pseudo app, which is quite cool, and only $15, but I figured it wouldn't be too hard to script something together. Well, Many Hours Later, I've got something! An AppleScript droplet that can be added to your dock and when you drag another App on it, a GUI popup will ask for your password, and voila! You have a GUI app running as root.

Open Applications..Utilities..AppleScriptEditor, paste this below, and then save it as MSudo.app, as an Application. Then drag it onto your Dock

Open

on open {filename}
     set p to POSIX path of filename
     set myArray to my theSplit(p, "/")
     set numItems to (count myArray) - 1
     set AppNameFull to getArrayValue(myArray, numItems)
     set myArray to my theSplit(AppNameFull, ".")
     set numItems to (count myArray) - 1
     set AppName to getArrayValue(myArray, 1)
     do shell script p & "/Contents/MacOS/" & AppName with administrator privileges
end open
on theSplit(theString, theDelimiter)
     -- save delimiters to restore old settings
     set oldDelimiters to AppleScript's text item delimiters
     -- set delimiters to delimiter to be used
set AppleScript's text item delimiters to theDelimiter
     -- create the array
     set theArray to every text item of theString
     -- restore the old setting
     set AppleScript's text item delimiters to oldDelimiters
     -- return the result
     return theArray
end theSplit
on getArrayValue(array, location)
     -- very important -- The list index starts at 1 not 0
     return item location in array
end getArrayValue

Compiling Erlang escript to .beam

2011-04-28T07:41:00.000-07:00

As part of my experiments in command line Erlang escripts, I've been looking into compiling escripts into .beam files. There is a bit of a wait on starting my .escript files, and command line tools should have as little delay on startup as possible since they are generally started and stopped often.

First, I compiled my .escript files to .beam, I did this with this bash script:

erlsee.sh (get it?):

#!/usr/bin/bash
test=$1
echo $test

args=("$@")
file="$1"
echo ${file}
temp_file=$1.tmp
echo $temp_file
cp $file $temp_file
file=`echo "$1" | sed s/.escript/.erl/g` 
sed -n '1!p' $temp_file > $file
/usr/bin/env erlc $file
cp $temp_file $file
rm $temp_file

This allows you to compile an escript directly without having to remove the #! directives at the top or rename the file to have an .erl ending.

Secondly, I added the same #! headers to the .beam files, and set execution permissions (`chmod u+x *.beam`). Just open the .beam file with a text editor and paste the following lines at the top:


#!/usr/bin/env escript
%%! -smp disable

Third, I tested the running times with a very small input file with the 'time' utility. I figure this time should be dominated by the basic start up time, not the execution speeds.

$ time cat test.txt | ./runavg.escript -s 5 -i 5 | ./tgraph.escript
************************************************************************24
*********************************************15

real 0m2.077s
user 0m0.121s
sys 0m0.045s

$ time cat test.txt | ./runavg.beam -s 5 -i 5 | ./tgraph.beam
************************************************************************24
*********************************************15

real 0m2.182s
user 0m0.090s
sys 0m0.138s

I ran this multiple times, and the averages were almost equal; certainly not a big improvment. So it's not worth the extra effort to compile your escript files to .beam to improve startup times.

Execution time is another matter; it is probably improved, but that doesn't really matter for programs that just output text. They are fast enough as is.

Running average in Erlang (escript)

2011-04-27T18:50:00.000-07:00

Another python util I've created a while back, and usually use in conjunction with tgraph, is a running (or moving) average util that can also be piped into. So, as another exercise, I created an Erlang/escript version.

The main use case is when I'm looking into a problem, and tailing a log. Too much data. Awk/perl/pyline the interesting numbers and tail that. Numbers running too fast or changing too often to see a pattern. So I then pipe the number stream through a running average and then to tgraph.

tail -f some.log | awk -F" " '{print $10}' | ./runavg.escript | ./tgraph.escript

Another similar case is to run a whole or portion of an existing log through runavg/tgraph to look for a pattern. Make sure you set your terminal's buffer size to 2-3K before trying that.

tail -10000 some.log | awk -F" " '{print $10}' | ./runavg.escript -s 10 -i 10 | ./tgraph.escript -t 1000

runavg.escript

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -smp disable
%% Author: Drew Gulino
-module(runavg).

-export([main/1]).

main(CmdLine) ->
OptSpecList = option_spec_list(),
case getopt:parse(OptSpecList, CmdLine) of
  {ok, {Options, NonOptArgs}} ->
    true;
  {error, {Reason, Data}} ->
    Options = [],
    NonOptArgs = [],
    io:format("Error: ~s ~p~n~n", [Reason, Data]),
    version(),
    getopt:usage(OptSpecList, "runavg")
end,
SampleSize = get_opt_value(size,Options),
OutputInterval = get_opt_value(interval,Options),

case NonOptArgs of
  [] ->
  F = standard_io;
  _ ->
  {ok, F} = file:open(NonOptArgs, read)
end,
%io:format("~p,~p,~p~n",[F, SampleSize, OutputInterval]),
proc_file(F, SampleSize, OutputInterval ).


proc_file(F, SampleSize, OutputInterval) ->
 %io:format("1"),
 proc_file(F, SampleSize, OutputInterval, [], 0, 0).

proc_file(F, SampleSize, OutputInterval, SampleAcc, SampleCount, IntervalCount) when IntervalCount >= OutputInterval ->
  io:format("~.2f~n",[lists:sum(SampleAcc)/erlang:length(SampleAcc)]),
  proc_file(F, SampleSize, OutputInterval, SampleAcc, SampleCount, 0);
proc_file(F, SampleSize, OutputInterval, [_|T] , SampleCount, IntervalCount) when SampleCount >= SampleSize ->
  %io:format("T: ~p~n",[T]),
  proc_file(F, SampleSize, OutputInterval, T, SampleCount - 1, IntervalCount);
proc_file(F, SampleSize, OutputInterval, SampleAcc, SampleCount, IntervalCount) ->
%io:format("SampleAcc: ~p~n", [SampleAcc]),
L = io:get_line(F, ''),
case L of
  eof ->
    ok;
  "\n" ->
    false;
  Line ->
    Stripped = strip_newlines(Line),
    Num = cast_to_integer(Stripped),
    proc_file(F, SampleSize, OutputInterval, [Num] ++ SampleAcc, SampleCount + 1, IntervalCount + 1)
end.

version() ->
  io:format("Version: 1.0\n").

get_opt_value(Key, Options) ->
  case lists:keyfind(Key,1,Options) of
    {Key, Value} ->
      Value
  end,
  Value.

option_spec_list() ->
[
%% {Name, ShortOpt, LongOpt, ArgSpec, HelpMsg}
{help, $h, "help", undefined, "Show the program options"},
{version, $v, "version", undefined, "Version"},
{size, $s, "size", {integer, 5}, "Size of average sample, Default=5"},
{interval, $i, "interval", {integer, 5}, "How many input entries before average is displayed, Default=5"}
].
       
strip_newlines(String) ->
string:strip(re:replace(String,"(.*)[\n\r]","\\1", [global,{return,list}])).

cast_to_integer([]) ->
[];
cast_to_integer(Input) when is_integer(Input) ->
Input;
cast_to_integer(Input) when is_float(Input) ->
erlang:round(Input);
cast_to_integer(Input) when is_list(Input)->
case lists:member($., Input) of
  true ->
    erlang:round(erlang:list_to_float(Input));
  false ->    
    erlang:list_to_integer(Input)
end.

A util I use to test numeric pipes is a random number streamer:
randstream.escript:

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -smp disable
%% Author: Drew Gulino
-module(randstream).

-export([main/1]).

main(_) ->
{A1,A2,A3} = now(),
random:seed(A1, A2, A3),
rand().

rand() ->
Num = random:uniform(100),
io:format("~B~n",[Num]),
rand().

./randstream.escript | ./runavg.escript -s 100 -i 100

should return a list of numbers right around 49 (average between 0 and 100).

Console plotting in Erlang

2011-04-25T21:26:00.000-07:00

Years ago I created a simple Python script to plot a list of numbers in a simple ASCII line graph. I use this script all the time; I extract (with awk,perl,pyliner,...) a column of numbers out of a log file and pipe it in to this script. Very useful since even to today most admin work is done in a character based terminal.

UPDATE (4/28/2011): Also check my other terminal console escript entries: runavg.escript, escript to beam

All numbers are rounded to an integer. It auto resizes new graph entries when a new max is set (default max is 0), and plots a new max line in bold. If you set a threshold, it outputs any line over that threshold in red. Or both.

I created an Erlang escript version of it, just as an exercise. It requires 'tput' to be in the path, this even works with the cygwin version. It also uses An Erlang version of getopt. I didn't bother to install it, I just copied getopt.erl to my src dir and compiled it. Also chmod u+x tgraph.escript

It's called as part of a pipe:
$ cat test.txt | ./tgraph.escript -t 40 -c 40

or with the file as a parameter:
./tgraph.escript test.txt -t 40 -c 40

test.txt:

Output:


****************************************1
****************************************50
****************20
**3
************************************45
***************************34
**********12
0
****************************************1000
0
****100
*34

tgraph.escript:

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -smp disable
%% Author: Drew Gulino
-module(tgraph).

-export([main/1]).

main(CmdLine) ->
 OptSpecList = option_spec_list(),
 case getopt:parse(OptSpecList, CmdLine) of
  {ok, {Options, NonOptArgs}} ->
   true;
  {error, {Reason, Data}} ->
   Options = [],
   NonOptArgs = [],
   io:format("Error: ~s ~p~n~n", [Reason, Data]),
   version(),
   getopt:usage(OptSpecList, "tgraph")
 end,
 Symbol = get_opt_value(symbol,Options),
 Columns = get_opt_value(columns,Options),
 Display_number = get_opt_value(display_number,Options),
 Threshold = get_opt_value(threshold,Options),
 Maximum = get_opt_value(maximum,Options),

Bold = strip_newlines(os:cmd("tput bold")),
 Init = strip_newlines(os:cmd("tput init")),
 Dim = strip_newlines(os:cmd("tput sgr0")),
 Red = strip_newlines(os:cmd("tput setaf 1")),
 %Green = strip_newlines(os:cmd("tput setaf 2")),
 %Yellow = strip_newlines(os:cmd("tput setaf 3")),
 %Blue = strip_newlines(os:cmd("tput setaf 4")),
 %Magenta = strip_newlines(os:cmd("tput setaf 5")),

 case NonOptArgs of
  [] -> 
  F = standard_io;
  _ ->
  {ok, F} = file:open(NonOptArgs, read)
 end,
  proc_file(F, {Symbol, Columns, Display_number, Threshold, Maximum} , {Bold, Init, Dim, Red}).

version() ->
 io:format("Version: 1.1\n").

get_opt_value(Key, Options) ->
 case lists:keyfind(Key,1,Options) of
  {Key, Value} ->
  Value
 end,
 Value.

option_spec_list() ->
 %CurrentUser = os:getenv("USER"),
 [
 %% {Name, ShortOpt, LongOpt, ArgSpec, HelpMsg}
 {help, $h, "help", undefined, "Show the program options"},
 {version, $v, "version", undefined, "Version"},
 {display_number, $n, "display_number", {boolean, true}, "Display number w/graph"},
 {columns, $c, "columns", {integer, 72}, "Display columns (default = 72)"},
 {symbol, $s, "symbol", {string, "*"}, "Symbol to display (default = '*')"},
 {threshold, $t, "threshold", {integer, 0}, "Will color lines over this value"},
 {maximum, $m, "maximum", {integer, 0}, "Presets the scale for this maximum value (default = 0)"}
 ].

proc_file(F, Options, Tput) ->

{Symbol, Columns, Display_number, Threshold, Maximum} = Options,

{Bold, Init, Dim, Red} = Tput,

%Columns = erlang:list_to_integer(os:cmd("tput cols")) - 8},

L = io:get_line(F, ''),
 case L of
  eof ->
   ok;
  "\n" ->
   false;
  Line ->
   Stripped = strip_newlines(Line),
   Num = cast_to_integer(Stripped),
   io:put_chars(Init),
   case Num > 0 of
    true ->
     case Num >= Maximum of
      true ->
       NewMax = Num,
       io:put_chars(Bold);
      false ->
       NewMax = Maximum,    
       io:put_chars(Dim)
     end,
     Scale = Columns / NewMax, 
     Graph = lists:map(fun(_) -> io_lib:format(Symbol,[]) end , lists:seq(1,erlang:round(Num * Scale))),
     case Threshold of
      0 -> 
       false;
      _ ->
       case Num >= Threshold of
        true ->
         io:put_chars(Red);
        false -> 
         %io:put_chars(Init)
         false
       end
     end,
     case Display_number of
      true ->
       io:format("~s~p~n",[Graph,Num]);
      false ->
       io:format("~p~n",[Graph])
     end,                                
     NewOptions = {Symbol, Columns, Display_number, Threshold, NewMax},
     proc_file(F,NewOptions, Tput);
    false ->
     io:put_chars(Dim),
     io:put_chars(Init),
     io:format("~p~n",[Num]),
     proc_file(F,Options, Tput)
   end
 end.

strip_newlines(String) ->
 string:strip(re:replace(String,"(.*)[\n\r]","\\1", [global,{return,list}])).

cast_to_integer([]) ->
 [];
cast_to_integer(Input) when is_integer(Input) ->
 Input;
cast_to_integer(Input) when is_float(Input) ->
 erlang:round(Input);
cast_to_integer(Input) when is_list(Input)->
 case lists:member($., Input) of
  true ->
   erlang:round(erlang:list_to_float(Input));
  false ->      
   erlang:list_to_integer(Input)
end.

UPDATE (4/26/2001):
Here's a link to the original python script: tgraph.py

UPDATE (4/27/2011):
Version 1.1:
1) Fixed bug where lines were never dimmed after being bolded in cygwin
2) Changed the compiler flags to disable smp and not register the process name. Both not needed. One note: Couldn't get +Bc to work in Windows. This should change the break key to Ctrl-C, but still stays Ctrl-Break.
3) Moved tput initialization out of working loop, now runs quickly.

iOS consolidated.db workaround for hacked devices

2011-04-20T07:19:00.000-07:00

Looks like Apple is tracking iOS devices an recording that info in clear text:
http://radar.oreilly.com/2011/04/apple-location-tracking.html

Here's a way to ensure this data is not recorded:

You must have a hacked iOS device, and either Mobile Terminal or an SSH login. You must also know the root password. You first remove/move this file, and recreate it as a symbolic link to /dev/null like:

su
cd /System/Library/Frameworks/CoreLocation.framework/Support
rm consolidated.db
ln -s /dev/null consolidated.db

Anything written to this 'file' is sent to /dev/null, so it is not saved on the file system. I've done this on a hacked device, and Location Services continue to work.

90th percentile done wrong

2011-04-17T17:37:00.000-07:00

I'm no statistician, but I do load test a lot of systems and report upon them.

It's common practice to report upon a particular measure using the average of the 90th percentile of the data (throwing out the slowest 10%): response times, throughput, etc. This is done to remove the outliers; the 2 days response time for a call that normally takes 100ms.

I got to thinking; if the slowest 10% is obviously wrong, why isn't the fastest 10%? Seems just taking the slow outliers is cheating.

I think a good compromise between simplicity and accuracy would be to throw out the slowest 5% and the fastest 5%.

A statistician would know better I'm sure. But explaining 90th percentile to your boss/customer is generally hard enough.

Erlang Hot Code Swapping - Interfaces

2011-04-15T21:16:00.000-07:00

I read Raymond Tay's 2007 post Erlang Hot Code Swapping a while back, and realized it was not quite right, or at least not complete. The idea and code behind code swapping works, but the execution of the code in the blog didn't actually demonstrate that fact. He changes the calling functions along with swapping the code; you shouldn't have to do that.

Which brings me to interfaces. EJBs (what he was trying to emulate) are all about interfaces. You specifically have to have separate files for the interface and the implementation. In fact, EJB 1.0 was way too heavy with the interfaces, and it was a pain to work with.

But the general idea of an interface is awesome. USB anyone? Back in the bad old days there was the RS-232 serial interface, which was woefully under-standardized, so there was no guarantee that plugging two RS-232 devices together would work (usually the opposite). You'd spend a lot of time trying different configurations to get things work (Remember 9600-N-8-1?).

Interfaces dictate a list of methods (or functions) that can always be expected to be implemented by a piece of code. This is great for long running code that must work with other systems. The implementation changes over time, but the developer knows if he/she (ok, most likely he, but only statistically) is going to change the interface, they will break compatibility with other systems.

The neat thing is that the interface is already defined in Erlang code that implements callbacks; they are just the exported functions that will make the callbacks. But the Java idea of an Interface is a list of methods that are separate from the implementation that can be share across implementations. Interfaces aren't a language feature of Erlang, but OTP behaviours are Interfaces (among other things): They require a list of exported functions.

What's the best way to implement an interface in Erlang without OTP? The best way I've figured is this: Move the public functions from the private callback functions into separate modules, and then import the public functions. This way two implementations that import the interface module will have to share the same interface.

First, the generic container code:

container.erl:

-module(container).
-export([start/1, rpc/2, swap_code/1]).

-include("callback.hrl").

start(Mod) ->
register(?SERVERNAME, spawn(fun() -> loop(?SERVERNAME, Mod, Mod:init()) end)).

swap_code(Mod) -> rpc(?SERVERNAME, {swap_code, Mod}).

%
% Standard code for abstracting the "RPC-call" layer
%
rpc(Name, Request) ->
    Name ! {self(), Request},
    receive
        {Name, Response} -> Response
    end.

%
% Standard code for looping and waiting for messages from clients
%
loop(Name, Mod, OldState) ->
    receive
        {From, {swap_code, NewCallbackMod}} ->
            From ! {Name, ack},
            loop(Name, NewCallbackMod, OldState);
        {From, Request} ->
            {Response, NewState} = Mod:handle(Request,OldState),
            From ! {Name, Response},
            loop(Name, Mod, NewState)
    end.

The registered server name is stored in an .hrl file included in the container and callback code:

callback.hrl:

-define(SERVERNAME, moneyserver).

callback.erl:

-module(callback).
-export([dollarToYen/1, yenToEuro/1]).
-include("callback.hrl").

-import(container, [rpc/2]).

%% client routines
dollarToYen(Dollars) -> rpc(?SERVERNAME, {convertToYen, Dollars}).
yenToEuro(Yen) -> rpc(?SERVERNAME, {convertToEuro, Yen}).

callback_impl.erl:

-module(callback_impl).
-export([init/0, handle/2]).
-import(container, [rpc/2]).
%% client routines
-import(callback, [dollarToYen/1,yenToEuro/1]).

%% callback routines
init() -> dict:new().

handle({convertToYen, Dollars}, Dict) -> { Dollars * 126, Dict};
handle({convertToEuro, Yen}, Dict) -> {Yen * 0.0077, Dict}.

callback_impl2.erl:

-module(callback_impl2).
-export([init/0, handle/2]).
-import(container, [rpc/2]).
%% client routines
-import(callback, [dollarToYen/1,yenToEuro/1]).

%% callback routines
init() -> dict:new().

handle({convertToYen, Dollars}, Dict) -> { Dollars * 126 * 126, Dict};
handle({convertToEuro, Yen}, Dict) -> {Yen * 0.0077 * 0.0077, Dict}.

How this works:

RUNTIME:


> c(container).
> c(callback).
> c(callback_impl).
> c(callback_impl2).

% our first implementation:
> container:start(callback_impl).
> callback:dollarToYen(1). 
126
> callback:yenToEuro(1).  
0.0077

> container:swap_code(callback_impl2).
ack
> callback:yenToEuro(1).              
5.929e-5
> callback:dollarToYen(1).            
15876

Same interface, same exact call, but hot swapped implementations!

Erlang auto reload code in (w)erl shell

2011-03-30T17:17:00.000-07:00

UPDATE: I found an existing, more mature implimentation of this idea:
http://www.rustyrazorblade.com/2010/12/erlang-code-auto-reloader/. Please disregard the rest.

I develop most of my code in eclipse, because most of my professional work deals with Java, and I develop in many different languages, including erlang.

One of the reasons I use and IDE instead of just and editor is auto compilation. This doesn't work out that great with a erl or werl shell because they don't auto reload code that is compiled outside of the shell.

I found a cool method to reload compiled code here, but it didn't auto reload; you still had to type a command to reload, so not much better than just running c().

So, I figured, like many problems, Erlang could solve this with a process. I was right!

1) One thing to note is that you have to exclude recompiling itself, because that kills that running process. And you'll have to update that code path manually. You could also use a relative path to os:getenv("PWD") if that makes sense in your env.

2) Also note that moving reload.erl into a seperate directory than the path mentioned in the code will not stop it from being reloaded; erlang stores all paths added via -pa as relative to your main path. Say you put reload.erl in ~/test and the rest of your code in ~/erlang, start erl in ~/test/src, and add ~/test/ebin to that code path:

$/home/user/erlang: erl -pa ../../test/ebin

the reload module would be loaded from:
/home/user/erlang/ebin/../../test/ebin

To start:
reload:start() (defaults to reload every 1 second)
or
reload:start(N) (reloads every N milliseconds)

To stop:
reload:stop() (yes, just throws an exception, could be nicer)

CODE:

%% Author: Drew Gulino: drew dot gulino at gmail dot com
%% Created: Mar 30, 2011
%% Description: Process to auto reload compiled code in shell

-module(reload).

%%
%% Exported Functions
%%
-export([start/0, start/1, stop/0, reloadAll/0]).

%% API Functions
start() ->
    start(1000).

start(Milliseconds) ->
    register(reload, spawn_link(fun() -> loop(Milliseconds) end)).

stop() ->
    throw(reload_stopped).

%% Local Functions
loop(Milliseconds) ->
    reloadAll(),
    timer:sleep(1000),
    loop(Milliseconds).

reload(M) ->
    code:purge(M),
    code:soft_purge(M),
    {module, M} = code:load_file(M),
    {ok, M}.

reloadAll() ->
    Modules = [M || {M, P} <- code:all_loaded(), is_list(P)
        andalso M =/= reload
        andalso string:str(P, "/home/drew/workspace/erlang/") > 0],
    [reload(M) || M <- Modules].

Erlang to stress test DNS

2007-08-14T23:09:00.000-07:00

I've been playing with Erlang and I thought it would be a great fit to create a highly scalable, distributed load generation tool.

I found the dns interface code at http://www.trapexit.org/index.php/Lookup_MX_Records_In_DNS

I've created a little program that will take a list of domains in a file, and a configurable number of processes. It will then iterate through each domain in the file and select a random process it will send the domain to be resolved by. Here's what I have as of yet:

usage:
create a list of domains in a file called queries.txt
1> Pids = dns:groups(N)
2> dns:group_lookup(Pids)

-module(dns).
-export([start/0, start_load/0, lookup/2, group/1,group/2, group_lookup/1]).

start() ->
spawn(fun loop/0).

start_load() ->
Pids = group(10),
group_lookup(Pids).

lookup(Pid, What) ->
rpc(Pid,What).

rpc(Pid, Request) ->
Pid ! {self(), Request},
receive
{Pid, Response} ->
Response
end.

loop() ->
receive
{From, Domain} ->
dnslookup:init( [{192,168,1,1}] ),
Addr2 = dnslookup:lookupa(lib:nonl(Domain)),
io:format("~p~n",[Addr2]),
From ! {self(), Addr2},
loop()
end.

group(K) ->
group(K, []).

group(0, Pids) -> Pids;
group(K, Pids) ->
group(K-1, [dns:start()| Pids]).

group_lookup(Pids) ->
Domains = dnslookup:readlines("queries.txt"),
dnslookup:init( [{192,168,1,1}] ),
[dns:lookup( lists:nth( random:uniform( length(Pids) ), Pids), Domain ) || Domain <- Domains].

-module(dnslookup).
-export([ readlines/1, queries_from_list/1, for_each_line_in_file/4, queries_direct_from_file/0, lookupmx/1, lookupa/1, lookupptr/1, init/1, findptr/1]). -include_lib("kernel/src/inet_dns.hrl"). init([NS | T]) ->
ok = inet_db:add_ns(NS),
init(T);
init([]) -> ok;
init(Nameserver) ->
ok = inet_db:add_ns(Nameserver).

print_list([]) ->
[];
print_list([Head | Rest]) ->
io:format("~w~n",[Head]),
print_list(Rest).

findmx( [#dns_rr{ type=?S_A, data = Addr } | _] ) -> Addr;
findmx( [_ | T ]) -> findmx(T);
findmx( _Other ) -> none.

finda( [#dns_rr{ type=?S_A, data = Addr } | _] ) -> Addr;
finda( [_ | T ]) -> finda(T);
finda( _Other ) -> none.

findptr( [Head | _] ) ->
%% io:format("~s~n",[element(9,Head)]),
%% io:formant("~s~n",[Head]),
[element(9,Head)];
%% findptr( [_ | T ]) -> findptr(T);
findptr( _Other ) -> none.

findheader( Head ) ->
io:format("~w~n",[is_tuple(Head)]),
io:format("~w~n",[Head]).

lookupmx(Domain) ->
case inet_res:nslookup(Domain,1,mx) of
{ error, Reply } -> Reply;
{ ok, #dns_rec{ anlist = Ans } } ->
%% io:format("MX Ans ~w,~n", [Ans]),
%% print_list(Ans),
findmx( Ans )
end.

lookupa(Domain) ->
%init([{68,105,28,12}]),
case inet_res:nslookup(Domain,1,a) of
{ error, Reply } -> Reply;
{ ok, #dns_rec{ anlist = Ans } } ->
%% print_list(Ans),
%%io:format("A Ans ~w,~n", [Ans]),
finda( Ans )
end.

%% IP address to domain name resolution
lookupptr(Domain) ->
case inet_res:nslookup(Domain,1,ptr) of
{ error, Reply } -> Reply;
{ ok, #dns_rec{ anlist = Ans, arlist = Ar, qdlist = Qd, nslist = Ns, header = H } } ->
%% print_list(Ans),
%% io:format("PTR Ans ~w,~n", [Ans]),
%% io:format("PTR Ar ~w,~n", [Ar]),
%% io:format("PTR Qd ~w,~n", [Qd]),
%% io:format("PTR Ns ~w,~n", [Ns]),
%% io:format("PTR H ~w,~n", [H]),
findptr(Ans)
end.

for_each_line_in_file(Name, Proc, Mode, Accum0) ->
{ok, Device} = file:open(Name, Mode),
for_each_line(Device, Proc, Accum0).

for_each_line(Device, Proc, Accum) ->
case io:get_line(Device, "") of
eof -> file:close(Device), Accum;
Line -> NewAccum = Proc(Line, Accum),
for_each_line(Device, Proc, NewAccum)
end.

queries_direct_from_file() ->
for_each_line_in_file("queries.txt",
fun(X, Count) -> io:format("~s~n",[lib:nonl(X)]),
io:format("~s~n",[lookupa(lib:nonl(X))]),
Count + 1 end, [read], 0).

queries_from_list([]) ->
[];
queries_from_list([Head | Rest]) ->
dnslookup:init( [{172,28,40,52}] ),
io:format("~s sent~n",[lib:nonl(Head)]),
A = dnslookup:lookupa(lib:nonl(Head)),
queries_from_list(Rest).

readlines(FileName) ->
{ok, Device} = file:open(FileName, [read]),
get_all_lines(Device, []).

get_all_lines(Device, Accum) ->
case io:get_line(Device, "") of
eof -> file:close(Device), lists:reverse(Accum);
Line -> get_all_lines(Device, [Line|Accum])
end.

Intro

2007-07-29T14:21:00.000-07:00

Well, it's taken some time, but I finally started a Blog.

Why? Why do more documentation than I already do?

One reason: Went to OSCON and was asked 'do you have a blog' one too many time.

I currently work as a System Engineer doing performance and scalability testing. Not QA! (Not that there's anything wrong with that). Every test I do is automated and I'm often part of the solution to any problem I find.

I work for a company with very strict SLAs and so we spend a lot of time and money ensuring our systems will perform at whole-web scales.

Although everything I test is web-centric, almost none of it is web pages. So many times I can't use test tools like LoadRunner to produce load. I work with Open Source tools or just create my own.

I work with a lot of different technologies and have access to a large amount of gear that other people probably don't have access to.

So here is my Blog on what I've either played with or tested professionally. It's sometimes hard to tell the difference.

OSCON 2007

2007-07-25T10:39:00.000-07:00

I attended OSCON for the first time this year.

I concentrated on Python and Java.
Java:
Grails and Tapestry show how they can do Rails on the JVM.

Python:
Django shows how it can do Rails w/Python. Is there nothing else new under the sun?

I had been looking into TurboGears, mainly because of Django's lack of a built AJAX support and the book out on it, but Simon Willison showed how easy it was to add-on. I had read about some issues w/the TurboGears team, but didn't want to believe it. But I was impressed by Django's presense at OSCON and didn't hear a word on TurboGears, and that there will be book on it soon.

- I also attended the Python 3000 (3.0) session w/Guido. 3.0 will not be completely backward compatible, and it looks like a bit of work for not a lot of gain; just a syntactic cleanup. The main benefit is to add Java-like Collections, which is nice.

I've been getting into Erlang lately, so I was really excited about the Erlang BoF:

I was the only person other than the presenter with any experience with `lang. Still I'm quite the nube. Learned a few cool things. I still have to repeat to myself "It a pattern match, not an assignment" over and over again when reading code.
I heard that the new Erlang book Programming Erlang:
Software for a Concurrent World was a real hit at the Powell's book stand. I was going to order it, but took advantage no shipping and no Oregon sales tax.

There was talk about how to redefining the Erlang standard library OTP from meaning "Open Telecom Platform" to something more web-centric. I suggest renaming Erlang to URLang ;) It's pronounced the same.

I recently did some stress testing of 64bit BIND 9.x. I had an existing Perl based testing suite, but it didn't meet all my needs: I needed incremental updates running while I ran a whole bunch'o queries.

I looked around to see what libraries were available; I normally use either Java or Python. I found good dns libs for each.

I then tried out the same in Erlang. I'll add a post about that soon.