[NTLUG:Discuss] Transport tool - opinions

[Kyle Davenport]

Hi Greg,

      No responses, huh.  What do you expect? - we don't all have server
farms in our basements.  And I'm sorry I have no experiences to offer, but
plenty of opinions! ; )

      Your query is just way too open-ended.  Virtually every problem that
might justify a cluster, has different requirements, and there is no one
best solution for all.

      Fast, sure, but is latency or bandwidth more important?  Or, for
example, I would NOT do RPC or Corba, because of the overhead, but ease of
use and greater functionality might be more important to you.  And then of
course, there's the question of budget.  SGI has a marvelous solution for
distributed threads in their new NUMA machine, the Altix3000, but the 64
processor Itanium2 box would set you back $1.2million.

      I have accumulated a bunch of links in anticipation of my own beowolf
cluster, but my threshold is <$200/node using the latest hardware.  (The
nForce2 all-in-one mb is close, but none have built-in Gbit ethernet like
my server)

Kyle Davenport

From: Greg Edwards <greg at nas-inet.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
I'm working on a project where I want to do concurrent processing over a
server farm.  I'm looking to do concurrency literally down to the
function level (using C) by passing the processing duties off to the
most responsive server available.  I've mentioned this idea here before.

What I'm looking for are experiences and opinions about the current
tools available for Linux to do the data passing between systems.  I'm
not looking to reinvent the wheel but I need to have something that is
fast and will not require jumping through a whole bunch of hoops to
code.  The application needs to be the focus, not the transport tools.

Currently I'm leaning toward either RPC or Corba.  I would like to use
off the shelf solutions for the data passing if possible.  If the
current solutions prove to be lacking in performance I can go back and
resolve that later.

I've looked at PVM and MPI but as with most high speed processing
solutions the models are parallel and clusters.  I need more of a
touring machine solution where the processing duties can start on box A,
get passed to whomever is available, and eventually return to the
originating function on box A when complete.

The concept of doing concurrency across a bank of computers does not
seem to have made any headway in the last 15 years.  Threads provided a
major breakthrough but I don't know of any extensions that take threads
beyond on a single box.  But, then again I may have missed something in
this area of R&D.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<