Erlang auto reload code in (w)erl shell

UPDATE: I found an existing, more mature implimentation of this idea:
http://www.rustyrazorblade.com/2010/12/erlang-code-auto-reloader/. Please disregard the rest.

I develop most of my code in eclipse, because most of my professional work deals with Java, and I develop in many different languages, including erlang.

One of the reasons I use and IDE instead of just and editor is auto compilation. This doesn't work out that great with a erl or werl shell because they don't auto reload code that is compiled outside of the shell.

I found a cool method to reload compiled code here, but it didn't auto reload; you still had to type a command to reload, so not much better than just running c().

So, I figured, like many problems, Erlang could solve this with a process. I was right!

1) One thing to note is that you have to exclude recompiling itself, because that kills that running process. And you'll have to update that code path manually. You could also use a relative path to os:getenv("PWD") if that makes sense in your env.

2) Also note that moving reload.erl into a seperate directory than the path mentioned in the code will not stop it from being reloaded; erlang stores all paths added via -pa as relative to your main path. Say you put reload.erl in ~/test and the rest of your code in ~/erlang, start erl in ~/test/src, and add ~/test/ebin to that code path:

$/home/user/erlang: erl -pa ../../test/ebin

the reload module would be loaded from:
/home/user/erlang/ebin/../../test/ebin

To start:
reload:start() (defaults to reload every 1 second)
or
reload:start(N) (reloads every N milliseconds)

To stop:
reload:stop() (yes, just throws an exception, could be nicer)

CODE:

%% Author: Drew Gulino: drew dot gulino at gmail dot com
%% Created: Mar 30, 2011
%% Description: Process to auto reload compiled code in shell

-module(reload).

%%
%% Exported Functions
%%
-export([start/0, start/1, stop/0, reloadAll/0]).

%% API Functions
start() ->
    start(1000).

start(Milliseconds) ->
    register(reload, spawn_link(fun() -> loop(Milliseconds) end)).

stop() ->
    throw(reload_stopped).

%% Local Functions
loop(Milliseconds) ->
    reloadAll(),
    timer:sleep(1000),
    loop(Milliseconds).

reload(M) ->
    code:purge(M),
    code:soft_purge(M),
    {module, M} = code:load_file(M),
    {ok, M}.

reloadAll() ->
    Modules = [M || {M, P} <- code:all_loaded(), is_list(P)
        andalso M =/= reload
        andalso string:str(P, "/home/drew/workspace/erlang/") > 0],
    [reload(M) || M <- Modules].

Erlang to stress test DNS

I've been playing with Erlang and I thought it would be a great fit to create a highly scalable, distributed load generation tool.

I found the dns interface code at http://www.trapexit.org/index.php/Lookup_MX_Records_In_DNS

I've created a little program that will take a list of domains in a file, and a configurable number of processes. It will then iterate through each domain in the file and select a random process it will send the domain to be resolved by. Here's what I have as of yet:

usage:
create a list of domains in a file called queries.txt
1> Pids = dns:groups(N)
2> dns:group_lookup(Pids)

-module(dns).
-export([start/0, start_load/0, lookup/2, group/1,group/2, group_lookup/1]).

start() ->
spawn(fun loop/0).

start_load() ->
Pids = group(10),
group_lookup(Pids).

lookup(Pid, What) ->
rpc(Pid,What).

rpc(Pid, Request) ->
Pid ! {self(), Request},
receive
{Pid, Response} ->
Response
end.

loop() ->
receive
{From, Domain} ->
dnslookup:init( [{192,168,1,1}] ),
Addr2 = dnslookup:lookupa(lib:nonl(Domain)),
io:format("~p~n",[Addr2]),
From ! {self(), Addr2},
loop()
end.

group(K) ->
group(K, []).

group(0, Pids) -> Pids;
group(K, Pids) ->
group(K-1, [dns:start()| Pids]).

group_lookup(Pids) ->
Domains = dnslookup:readlines("queries.txt"),
dnslookup:init( [{192,168,1,1}] ),
[dns:lookup( lists:nth( random:uniform( length(Pids) ), Pids), Domain ) || Domain <- Domains].

-module(dnslookup).
-export([ readlines/1, queries_from_list/1, for_each_line_in_file/4, queries_direct_from_file/0, lookupmx/1, lookupa/1, lookupptr/1, init/1, findptr/1]). -include_lib("kernel/src/inet_dns.hrl"). init([NS | T]) ->
ok = inet_db:add_ns(NS),
init(T);
init([]) -> ok;
init(Nameserver) ->
ok = inet_db:add_ns(Nameserver).

print_list([]) ->
[];
print_list([Head | Rest]) ->
io:format("~w~n",[Head]),
print_list(Rest).

findmx( [#dns_rr{ type=?S_A, data = Addr } | _] ) -> Addr;
findmx( [_ | T ]) -> findmx(T);
findmx( _Other ) -> none.

finda( [#dns_rr{ type=?S_A, data = Addr } | _] ) -> Addr;
finda( [_ | T ]) -> finda(T);
finda( _Other ) -> none.

findptr( [Head | _] ) ->
%% io:format("~s~n",[element(9,Head)]),
%% io:formant("~s~n",[Head]),
[element(9,Head)];
%% findptr( [_ | T ]) -> findptr(T);
findptr( _Other ) -> none.

findheader( Head ) ->
io:format("~w~n",[is_tuple(Head)]),
io:format("~w~n",[Head]).

lookupmx(Domain) ->
case inet_res:nslookup(Domain,1,mx) of
{ error, Reply } -> Reply;
{ ok, #dns_rec{ anlist = Ans } } ->
%% io:format("MX Ans ~w,~n", [Ans]),
%% print_list(Ans),
findmx( Ans )
end.

lookupa(Domain) ->
%init([{68,105,28,12}]),
case inet_res:nslookup(Domain,1,a) of
{ error, Reply } -> Reply;
{ ok, #dns_rec{ anlist = Ans } } ->
%% print_list(Ans),
%%io:format("A Ans ~w,~n", [Ans]),
finda( Ans )
end.

%% IP address to domain name resolution
lookupptr(Domain) ->
case inet_res:nslookup(Domain,1,ptr) of
{ error, Reply } -> Reply;
{ ok, #dns_rec{ anlist = Ans, arlist = Ar, qdlist = Qd, nslist = Ns, header = H } } ->
%% print_list(Ans),
%% io:format("PTR Ans ~w,~n", [Ans]),
%% io:format("PTR Ar ~w,~n", [Ar]),
%% io:format("PTR Qd ~w,~n", [Qd]),
%% io:format("PTR Ns ~w,~n", [Ns]),
%% io:format("PTR H ~w,~n", [H]),
findptr(Ans)
end.

for_each_line_in_file(Name, Proc, Mode, Accum0) ->
{ok, Device} = file:open(Name, Mode),
for_each_line(Device, Proc, Accum0).

for_each_line(Device, Proc, Accum) ->
case io:get_line(Device, "") of
eof -> file:close(Device), Accum;
Line -> NewAccum = Proc(Line, Accum),
for_each_line(Device, Proc, NewAccum)
end.


queries_direct_from_file() ->
for_each_line_in_file("queries.txt",
fun(X, Count) -> io:format("~s~n",[lib:nonl(X)]),
io:format("~s~n",[lookupa(lib:nonl(X))]),
Count + 1 end, [read], 0).

queries_from_list([]) ->
[];
queries_from_list([Head | Rest]) ->
dnslookup:init( [{172,28,40,52}] ),
io:format("~s sent~n",[lib:nonl(Head)]),
A = dnslookup:lookupa(lib:nonl(Head)),
queries_from_list(Rest).


readlines(FileName) ->
{ok, Device} = file:open(FileName, [read]),
get_all_lines(Device, []).

get_all_lines(Device, Accum) ->
case io:get_line(Device, "") of
eof -> file:close(Device), lists:reverse(Accum);
Line -> get_all_lines(Device, [Line|Accum])
end.

Intro

Well, it's taken some time, but I finally started a Blog.

Why? Why do more documentation than I already do?

One reason: Went to OSCON and was asked 'do you have a blog' one too many time.

I currently work as a System Engineer doing performance and scalability testing. Not QA! (Not that there's anything wrong with that). Every test I do is automated and I'm often part of the solution to any problem I find.

I work for a company with very strict SLAs and so we spend a lot of time and money ensuring our systems will perform at whole-web scales.

Although everything I test is web-centric, almost none of it is web pages. So many times I can't use test tools like LoadRunner to produce load. I work with Open Source tools or just create my own.

I work with a lot of different technologies and have access to a large amount of gear that other people probably don't have access to.

So here is my Blog on what I've either played with or tested professionally. It's sometimes hard to tell the difference.

OSCON 2007

I attended OSCON for the first time this year.

I concentrated on Python and Java.
Java:
Grails and Tapestry show how they can do Rails on the JVM.

Python:
Django shows how it can do Rails w/Python. Is there nothing else new under the sun?

I had been looking into TurboGears, mainly because of Django's lack of a built AJAX support and the book out on it, but Simon Willison showed how easy it was to add-on. I had read about some issues w/the TurboGears team, but didn't want to believe it. But I was impressed by Django's presense at OSCON and didn't hear a word on TurboGears, and that there will be book on it soon.

- I also attended the Python 3000 (3.0) session w/Guido. 3.0 will not be completely backward compatible, and it looks like a bit of work for not a lot of gain; just a syntactic cleanup. The main benefit is to add Java-like Collections, which is nice.

I've been getting into Erlang lately, so I was really excited about the Erlang BoF:

I was the only person other than the presenter with any experience with `lang. Still I'm quite the nube. Learned a few cool things. I still have to repeat to myself "It a pattern match, not an assignment" over and over again when reading code.
I heard that the new Erlang book Programming Erlang:
Software for a Concurrent World was a real hit at the Powell's book stand. I was going to order it, but took advantage no shipping and no Oregon sales tax.

There was talk about how to redefining the Erlang standard library OTP from meaning "Open Telecom Platform" to something more web-centric. I suggest renaming Erlang to URLang ;) It's pronounced the same.

I recently did some stress testing of 64bit BIND 9.x. I had an existing Perl based testing suite, but it didn't meet all my needs: I needed incremental updates running while I ran a whole bunch'o queries.

I looked around to see what libraries were available; I normally use either Java or Python. I found good dns libs for each.

I then tried out the same in Erlang. I'll add a post about that soon.