Technical Musings: 2007

Tuesday, August 14, 2007

Erlang to stress test DNS

I've been playing with Erlang and I thought it would be a great fit to create a highly scalable, distributed load generation tool.

I found the dns interface code at http://www.trapexit.org/index.php/Lookup_MX_Records_In_DNS

I've created a little program that will take a list of domains in a file, and a configurable number of processes. It will then iterate through each domain in the file and select a random process it will send the domain to be resolved by. Here's what I have as of yet:

usage:
create a list of domains in a file called queries.txt
1> Pids = dns:groups(N)
2> dns:group_lookup(Pids)

-module(dns).
-export([start/0, start_load/0, lookup/2, group/1,group/2, group_lookup/1]).

start() ->
spawn(fun loop/0).

start_load() ->
Pids = group(10),
group_lookup(Pids).

lookup(Pid, What) ->
rpc(Pid,What).

rpc(Pid, Request) ->
Pid ! {self(), Request},
receive
{Pid, Response} ->
Response
end.

loop() ->
receive
{From, Domain} ->
dnslookup:init( [{192,168,1,1}] ),
Addr2 = dnslookup:lookupa(lib:nonl(Domain)),
io:format("~p~n",[Addr2]),
From ! {self(), Addr2},
loop()
end.

group(K) ->
group(K, []).

group(0, Pids) -> Pids;
group(K, Pids) ->
group(K-1, [dns:start()| Pids]).

group_lookup(Pids) ->
Domains = dnslookup:readlines("queries.txt"),
dnslookup:init( [{192,168,1,1}] ),
[dns:lookup( lists:nth( random:uniform( length(Pids) ), Pids), Domain ) || Domain <- Domains].




-module(dnslookup).
-export([ readlines/1, queries_from_list/1, for_each_line_in_file/4, queries_direct_from_file/0, lookupmx/1, lookupa/1, lookupptr/1, init/1, findptr/1]). -include_lib("kernel/src/inet_dns.hrl"). init([NS | T]) ->
ok = inet_db:add_ns(NS),
init(T);
init([]) -> ok;
init(Nameserver) ->
ok = inet_db:add_ns(Nameserver).

print_list([]) ->
[];
print_list([Head | Rest]) ->
io:format("~w~n",[Head]),
print_list(Rest).

findmx( [#dns_rr{ type=?S_A, data = Addr } | _] ) -> Addr;
findmx( [_ | T ]) -> findmx(T);
findmx( _Other ) -> none.

finda( [#dns_rr{ type=?S_A, data = Addr } | _] ) -> Addr;
finda( [_ | T ]) -> finda(T);
finda( _Other ) -> none.

findptr( [Head | _] ) ->
%% io:format("~s~n",[element(9,Head)]),
%% io:formant("~s~n",[Head]),
[element(9,Head)];
%% findptr( [_ | T ]) -> findptr(T);
findptr( _Other ) -> none.

findheader( Head ) ->
io:format("~w~n",[is_tuple(Head)]),
io:format("~w~n",[Head]).

lookupmx(Domain) ->
case inet_res:nslookup(Domain,1,mx) of
{ error, Reply } -> Reply;
{ ok, #dns_rec{ anlist = Ans } } ->
%% io:format("MX Ans ~w,~n", [Ans]),
%% print_list(Ans),
findmx( Ans )
end.

lookupa(Domain) ->
%init([{68,105,28,12}]),
case inet_res:nslookup(Domain,1,a) of
{ error, Reply } -> Reply;
{ ok, #dns_rec{ anlist = Ans } } ->
%% print_list(Ans),
%%io:format("A Ans ~w,~n", [Ans]),
finda( Ans )
end.

%% IP address to domain name resolution
lookupptr(Domain) ->
case inet_res:nslookup(Domain,1,ptr) of
{ error, Reply } -> Reply;
{ ok, #dns_rec{ anlist = Ans, arlist = Ar, qdlist = Qd, nslist = Ns, header = H } } ->
%% print_list(Ans),
%% io:format("PTR Ans ~w,~n", [Ans]),
%% io:format("PTR Ar ~w,~n", [Ar]),
%% io:format("PTR Qd ~w,~n", [Qd]),
%% io:format("PTR Ns ~w,~n", [Ns]),
%% io:format("PTR H ~w,~n", [H]),
findptr(Ans)
end.

for_each_line_in_file(Name, Proc, Mode, Accum0) ->
{ok, Device} = file:open(Name, Mode),
for_each_line(Device, Proc, Accum0).

for_each_line(Device, Proc, Accum) ->
case io:get_line(Device, "") of
eof -> file:close(Device), Accum;
Line -> NewAccum = Proc(Line, Accum),
for_each_line(Device, Proc, NewAccum)
end.


queries_direct_from_file() ->
for_each_line_in_file("queries.txt",
fun(X, Count) -> io:format("~s~n",[lib:nonl(X)]),
io:format("~s~n",[lookupa(lib:nonl(X))]),
Count + 1 end, [read], 0).

queries_from_list([]) ->
[];
queries_from_list([Head | Rest]) ->
dnslookup:init( [{172,28,40,52}] ),
io:format("~s sent~n",[lib:nonl(Head)]),
A = dnslookup:lookupa(lib:nonl(Head)),
queries_from_list(Rest).


readlines(FileName) ->
{ok, Device} = file:open(FileName, [read]),
get_all_lines(Device, []).

get_all_lines(Device, Accum) ->
case io:get_line(Device, "") of
eof -> file:close(Device), lists:reverse(Accum);
Line -> get_all_lines(Device, [Line|Accum])
end.

Sunday, July 29, 2007

Intro

Well, it's taken some time, but I finally started a Blog.

Why? Why do more documentation than I already do?

One reason: Went to OSCON and was asked 'do you have a blog' one too many time.

I currently work as a System Engineer doing performance and scalability testing. Not QA! (Not that there's anything wrong with that). Every test I do is automated and I'm often part of the solution to any problem I find.

I work for a company with very strict SLAs and so we spend a lot of time and money ensuring our systems will perform at whole-web scales.

Although everything I test is web-centric, almost none of it is web pages. So many times I can't use test tools like LoadRunner to produce load. I work with Open Source tools or just create my own.

I work with a lot of different technologies and have access to a large amount of gear that other people probably don't have access to.

So here is my Blog on what I've either played with or tested professionally. It's sometimes hard to tell the difference.

Wednesday, July 25, 2007

OSCON 2007

I attended OSCON for the first time this year.

I concentrated on Python and Java.
Java:
Grails and Tapestry show how they can do Rails on the JVM.

Python:
Django shows how it can do Rails w/Python. Is there nothing else new under the sun?

I had been looking into TurboGears, mainly because of Django's lack of a built AJAX support and the book out on it, but Simon Willison showed how easy it was to add-on. I had read about some issues w/the TurboGears team, but didn't want to believe it. But I was impressed by Django's presense at OSCON and didn't hear a word on TurboGears, and that there will be book on it soon.

- I also attended the Python 3000 (3.0) session w/Guido. 3.0 will not be completely backward compatible, and it looks like a bit of work for not a lot of gain; just a syntactic cleanup. The main benefit is to add Java-like Collections, which is nice.

I've been getting into Erlang lately, so I was really excited about the Erlang BoF:

I was the only person other than the presenter with any experience with `lang. Still I'm quite the nube. Learned a few cool things. I still have to repeat to myself "It a pattern match, not an assignment" over and over again when reading code.
I heard that the new Erlang book Programming Erlang:
Software for a Concurrent World
was a real hit at the Powell's book stand. I was going to order it, but took advantage no shipping and no Oregon sales tax.

There was talk about how to redefining the Erlang standard library OTP from meaning "Open Telecom Platform" to something more web-centric. I suggest renaming Erlang to URLang ;) It's pronounced the same.

I recently did some stress testing of 64bit BIND 9.x. I had an existing Perl based testing suite, but it didn't meet all my needs: I needed incremental updates running while I ran a whole bunch'o queries.

I looked around to see what libraries were available; I normally use either Java or Python. I found good dns libs for each.

I then tried out the same in Erlang. I'll add a post about that soon.