[ANN] Drake: Distributed Rake

ruby-talk
21

Thomas Sawyer wrote:

in that order. �However, that doesn't provide any guarantees that they
(its just more likely that the drake will make this kind of bug
manifest).

Ah, so by design you consider it a bug. You could have fixed that from
day one by randomizing the order of the prerequisites. Now you have a
situation where many Rakefiles depend on that bug. So, why not turn
lemons into lemonade, and make this bug a feature?

I'm not sure what you are advocating here:

(1) Guarantee that rake will invoke the prerequisites in the defined
order? ... we already do that (for standard non-tasking rake).

(2) Guarantee that rake will execute the prerequisites in the defined
order? ... Can't do that, prerequisite constraints elsewhere may
constrain the execution to be a different order.

(3) Declare that I don't mind if you make unwarranted assumptions about
execution order? ... Well, as long as you don't file bug reports, I'm ok
with that.

More clarification on Rake terminology:

To execute a task means to execute any code blocks attached to the task
(i.e. the do/end part of a task).

To invoke a task means to make sure all the prerequisites for the task
have been invoked and then execute the task if it has not yet been
executed. A task invocation will not execute the task if it has already
been executed.

In standard rake, the order of dependencies only specifies the
invocation order, not the execution order. You never were able to
directly control execution order of tasks via the order of the
dependency list.

In moving to drake, what you lose is direct control over invocation
order. You never had direct control of execution order.

-- Jim Weirich

···

On Sep 10, 7:59�am, Jim Weirich <j...@weirichhouse.org> wrote:

--
Posted via http://www.ruby-forum.com/\.

22

There is a mathematical reality we cannot avoid, from which
special-case syntax and backwards-compatibility acrobatics cannot save
us. The problem is in our thinking. We didn't specify what depends
on what. We thought we did, but it turns out we were fooling
ourselves all along.

That's not always the problem. Given that Rake itself doesn't guarantee any
kind of ordering, we have to assume that dependencies are specified
correctly, or close to it.

But we're not writing Erlang, which means spec-ing dependencies correctly
isn't enough.

Trans suggested that this

    task :a => [:x, :y, :z]

should be translated into this

    task :a => :z
    task :z => :y
    task :y => :x

while this

    task :a => [[:x, :y, :z]]

is translated into this

    task :a => :x
    task :a => :y
    task :a => :z

OK, but there are a million ways in which a programmer can
insufficiently define dependencies. This will not come close to
saving us.

No, but it does take us back to the behavior of Rake, or of Drake -j1. If you
really want to provide bug-for-bug compatibility, dig into the Rake code and
figure out what the ordering should be.

There is already a historical precedent with Makefiles. A new syntax
could have been added to Makefiles, but none was. The Makefiles had
bugs, but instead of timidly skirting around the problems while
praising the gods of backwards compatibility, people faced them
head-on, solving them one at at time.

Some did, yes.

And some let their Makefiles remain, with the existing syntax and bugs, and
left it to their users to figure out whether they could be parellized or not.

I'm sorry, but if you're already asking me to manually run a rake task, you
don't get to also ask me to read the source code of your Rakefile and figure
out whether or not it will work with -j2. Nor should I have to use trial and
error, potentially with very subtle bugs, to figure out what's happened.

And it's worth mentioning again: We're not writing Erlang, we're writing Ruby.

That means shared memory. It means locking issues. And it means thread-unsafe
libraries.

It means that a Rakefile could very well crash if run with -j2.

Understand, I don't mean it will be run in the wrong order, or that the
dependencies are wrong. The dependencies may well be perfect, and it will run
exactly as designed to.

Except that at some point, two separate tasks will simultaneously do something
a library won't like, and that library will deadlock. Or segfault. Or worse,
give corrupt data.

Which means that the Rakefile author is responsible, then, for fixing the
deficiencies in the library. Or they have to contact the library author, and
attempt to get the library fixed. Making every single Ruby library
thread-safe is a laudable goal, but also not going to happen.

You could solve a lot of that, I suppose, by forking instead -- but that
introduces its own problems.

···

On Tuesday 09 September 2008 17:38:40 . wrote:

23

> > So what do you do with things which aren't thread-save? Or
> > "j-safe"?

> The same thing you do with a Makefile that isn't j-safe: (1) write
> the dependencies correctly, which makes it j-safe, or (2) don't
> run it with -j.

Still going to be a fair number of cases of (3), I imagine: use
locks to synchronize non-thread-safe libraries, for which there's
still a benefit to running those tasks in parallel.

If by "non-thread-safe libraries" you mean a library whose Rakefile is
not j-safe, then you would just run it without -j. If it is a library
inside a larger project, you have at least two options:

(a) Run single-threaded rake in a subprocess for that library.

(b) Use the Rake module directly, as the unit tests do. The
no-invoke-inside-invoke rule applies per TaskManager, so you could
create a new TaskManager and do whatever you wish with it.

Keep in mind this is supposed to be stop-gap as you wait for the
author of the Rakefile in question to write the dependencies
correctly.

> If 'task' became 'multitask', Rake would run all your tasks at
> once -- all at the same time. That's probably not what you want
> :slight_smile:

Actually, no, I assumed that 'multitask' only ran that specific task
in parallel.

Actually, I hadn't thought about it thoroughly enough to realize
that this wasn't what was happening:

> The dependency graph tells us what can be run in parallel and what
> can't.

I understand make -j, and I think I understand the difference with
multitask -- if I understand it:

multitask :foo ...
multitask :bar ...

In the above example, will everything really run concurrently? I'd
assumed that foo would run concurrently, and then bar would run
concurrently.

When I mentioned that multitask is an alias of task, you asked about
the converse: what if task is an alias of multitask? multitask
blindly executes all its prereqs at once, in parallel, one task per
thread. So if everything is a multitask, applying the rule
recursively, all tasks run at once.

Getting back to your question, your code

multitask :foo ...
multitask :bar ...

does not say "do foo, then do bar" (whether it's a task or a multitask
is not relevant). The code is only defining the dependency relations
and the tasks associated with each node, to be executed later. If foo
lists bar as a prereq, then bar will run before foo.

In either case, I see what Drake is doing (real make -j
behavior). Thanks for explaining this -- it looks cool!

One more thing: I'm not sure what the best way to do this is, but I
think it would still be useful to have the task/multitask dichotomy,
for legacy programs. Multitasks would operate as properly parallized
Drake tasks. Plain old tasks would run in complete isolation, with
the exception that if they invoke a multitask, that multitask (and
all its remaining dependencies) run in j-parallized mode.

I had anticipated the suggestion that 'multitask' enable parallelism.
It cannot solve the problem. See my other post on superstructures
around the graph concept.

As I mentioned, drake is 100% compatible with rake for -j1, the
default behavior. The only thing we lose is the speedup for a -j1
multitask, which is recovered (and then some) once the dependencies
are written correctly and -jN may be run.

I don't think it is too much to ask: in exchange for fixing your
dependency logic, your build times will divided by N (best case) for
an N-core machine. If you don't fix it, then expect a little slowdown
if you were using multitask for something nontrival. That situation
seems rare enough, and harmless enough, to not worry much about.

Also, do we really want to create elaborate special cases, introducing
bugs and complexity, just to save someone a few moments (perhaps
nothing) during a build? Especially considering this is being done in
order to allow the person to retain the logic errors in his code?

That would certainly break the purity of it, and it would be a bit
more work, but I think it could be made to work. The benefit is, you
could translate an existing project iteratively, without having to
verify that the whole thing is correct, first.

You can already translate it piecemeal, if that is what you meant by
iteratively. It simply requires testing the bottom-level tasks
individually and working your way up. When I explained this
previously, I mistakenly said bottom-level-last-executed. I meant the
opposite, of course -- bottom level (most-depended-upon) is executed
first.

···

On Tuesday 09 September 2008 05:03:43 quixoticsycoph...@gmail.com wrote:
> On Sep 9, 4:52 am, David Masover <ni...@slaphack.com> wrote:

24

Yes, however my point was that if we agree this

    task :a => :x
    task :a => :y
    task :a => :z

should be equivalent to this

    task :a => :z
    task :a => :x
    task :a => :y

(and I should hope we all agree), then there is nothing which can save
us. We are forced to write our Rakefiles correctly. No backwards
compatibility mode is possible for threads>1.

JL

···

On Sep 9, 6:52 pm, Martin DeMello <martindeme...@gmail.com> wrote:

On Tue, Sep 9, 2008 at 3:38 PM, . <quixoticsycoph...@gmail.com> wrote:

> Do we want

> task :a => :x
> task :a => :y
> task :a => :z

> to be equivalent to

> task :a => :z
> task :a => :x
> task :a => :y

> Yes or no? If a programmer wants them to mean different things, how
> shall we accommodate him?

Yes! If you want it differently, you write in the ordering explicitly

task :a => :z
task :z => :y
task :y => :x

You have no reason to expect one operation to come before another if
there is not an explicit dependency chain between them

martin

25

David Masover wrote:

Actually, no, I assumed that 'multitask' only ran that specific task in
parallel.

Actually, multitask will run all of the tasks dependencies in parallel,
not the task itself.

-- Jim Weirich

···

--
Posted via http://www.ruby-forum.com/\.

26

And yet we can use the execution order in practice:

  F =
  G =

  task :f do
    F.replace([1,2,3])
  end

  task :g do
    if F.empty?
      G.replace([4,5,6])
    else
      G.replace(F)
    end
  end

  desc "use f and g not defined by f"
  task :g1 => [:g, :f] do
    p G, F
  end

  desc "use f and g defined by f"
  task :g2 => [:f, :g] do
    p G, F
  end

I understand that the formal design did not intend for this. But
implementation allows it.

Is it worth potentially breaking Rakefiles to prevent this sort of
thing (like drake -j2 or more does)? I'm not so sure. While one might
consider this Rakefile "bad design" because it doesn't fit the
original formal notion, it nonetheless does what one would expect it
to do. I think I'd rather have that, than the potential for ambiguous
behavior.

T.

···

On Sep 10, 9:13 am, Jim Weirich <j...@weirichhouse.org> wrote:

Thomas Sawyer wrote:
> On Sep 10, 7:59 am, Jim Weirich <j...@weirichhouse.org> wrote:
>> in that order. However, that doesn't provide any guarantees that they
>> (its just more likely that the drake will make this kind of bug
>> manifest).

> Ah, so by design you consider it a bug. You could have fixed that from
> day one by randomizing the order of the prerequisites. Now you have a
> situation where many Rakefiles depend on that bug. So, why not turn
> lemons into lemonade, and make this bug a feature?

I'm not sure what you are advocating here:

(1) Guarantee that rake will invoke the prerequisites in the defined
order? ... we already do that (for standard non-tasking rake).

(2) Guarantee that rake will execute the prerequisites in the defined
order? ... Can't do that, prerequisite constraints elsewhere may
constrain the execution to be a different order.

(3) Declare that I don't mind if you make unwarranted assumptions about
execution order? ... Well, as long as you don't file bug reports, I'm ok
with that.

More clarification on Rake terminology:

To execute a task means to execute any code blocks attached to the task
(i.e. the do/end part of a task).

To invoke a task means to make sure all the prerequisites for the task
have been invoked and then execute the task if it has not yet been
executed. A task invocation will not execute the task if it has already
been executed.

In standard rake, the order of dependencies only specifies the
invocation order, not the execution order. You never were able to
directly control execution order of tasks via the order of the
dependency list.

27

James M. Lawrence wrote:
> If 'task' became 'multitask', Rake would run all your tasks at once --
> all at the same time. That's probably not what you want :slight_smile:

> [...] It's a math problem.
> 'multitask' stomps it all to pieces, having the power to declare 2 + 5
> = 8 if it so chooses.

I'm not quite sure what you are saying here, but if you are trying to
imply that multitask does not honor dependencies in ordering, you are
incorrect. If there is dependency declared, then a task won't run until
all of its dependencies have finished.

That being said, there is a known bug in multitask where failures in
dependencies are not properly transmitted to all dependent tasks. But I
don't think you were refering to that.

I did not mean to imply there was something wrong with standard Rake.
It was being argued that Drake should retain the 'multitask' feature
for backwards compatibility. But from the point of view of Drake,
'multitask' is a mistake, the antithesis of everything Drake tries to
achieve. At a given point during execution, what is parallelizable is
a math problem with only one answer. But multitask comes prancing in
and announces: "I'm going to parallelize you, you, and you, just
because it's my birthday and I'm in a good mood." And then Drake is
like, "WTF are you doing? Not only is that the wrong answer to the
math problem, you're also trashing the node-locking algorithms."

If 'task' became 'multitask', Rake would run all your tasks at once --
all at the same time. That's probably not what you want :slight_smile:

Now that I read my own words here, I see it is misleading. I should
have said: If 'task' became 'multitask', Rake would run all your tasks
at once, one per thread all at the same time, but each thread would
still block until prereqs are filled. Which *could* be what you want
on a small project, but even a small project can have enough tasks to
bog it down.

Looking back at the beginning of this thread, I did not mean to imply
Jim moved to github just for me. He was moving anyway. Due to my
asking for a branch commit in the SVN repository, he may have moved a
little sooner. Which was convenient for me. Thanks.

JL

···

On Sep 10, 8:19 am, Jim Weirich <j...@weirichhouse.org> wrote:

28

Jos Backus wrote:

···

On Wed, Sep 10, 2008 at 08:38:35AM +0900, . wrote:

On Sep 9, 11:25�am, Jos Backus <j...@catnook.com> wrote:
> Does Drake properly clean up its children if it is aborted with SIGINT? ISTR
> multitask in rake leaving orphans running.

I'm misremembering. SIGINT seems to work okay, it's SIGTERM that leaves
orphaned children (with ppid 1) around with rake, presumably because it
doesn't catch that signal. Same with drake (0.8.1.11.0.1)

What would be a good way to fix this?

-- Jim Weirich
--
Posted via http://www.ruby-forum.com/\.

29

A --file-order-implies-dependency flag might get us there in a lot of
cases, though of course there's no general solution. Of more value
would be a lint tool that helps convert a rakefile into parallelisable
form.

martin

···

On Tue, Sep 9, 2008 at 4:13 PM, . <quixoticsycophant@gmail.com> wrote:

(and I should hope we all agree), then there is nothing which can save
us. We are forced to write our Rakefiles correctly. No backwards
compatibility mode is possible for threads>1.

30

They aren't potentially broken, they are broken. If it happens to work, you've just gotten lucky.

I've helped rework the Rubinius rakefiles twice and I can assure you it's perfectly possible to having broken rakefiles without -j. We were able to use drake with only one change due to having working rakefiles beforehand.

Furthermore, this is a feature that is not enabled by default. I don't see where this is an issue.

···

On Sep 10, 2008, at 11:06 AM, Trans wrote:

Is it worth potentially breaking Rakefiles to prevent this sort of
thing (like drake -j2 or more does)? I'm not so sure. While one might
consider this Rakefile "bad design" because it doesn't fit the
original formal notion, it nonetheless does what one would expect it
to do. I think I'd rather have that, than the potential for ambiguous
behavior.

31

While one might consider this Rakefile "bad design" because it
doesn't fit the original formal notion, it nonetheless does what one
would expect it to do. I think I'd rather have that, than the
potential for ambiguous behavior.

Underspecified dependencies + parallel execution == ambiguous behavior

There's no such thing as luck in computer programming.

Yes, there is.

···

On Sep 10, 2:06 pm, Trans <transf...@gmail.com> wrote:

32

> Still going to be a fair number of cases of (3), I imagine: use
> locks to synchronize non-thread-safe libraries, for which there's
> still a benefit to running those tasks in parallel.

If by "non-thread-safe libraries" you mean a library whose Rakefile is
not j-safe, then you would just run it without -j.

Which is, by the way, one of the most irritating things about Makefiles.

While in the simple case, a Makefile author might not know about -j, and write
a safe Makefile anyway (because that's really simpler, after all), it's
really troublesome that there's no standard way to tell whether something's
j-safe or not.

Seems like the best I can do is run something with -j, and if it seems to
work, well, hope for the best.

That's one reason I like multitask -- it forces the programmer to be
explicitly thinking about threading.

If it is a library
inside a larger project, you have at least two options:

(a) Run single-threaded rake in a subprocess for that library.

(b) Use the Rake module directly, as the unit tests do. The
no-invoke-inside-invoke rule applies per TaskManager, so you could
create a new TaskManager and do whatever you wish with it.

I was talking about an even simpler problem:

Let's pretend, for a moment, that we're talking to an HTTP library that's not
thread-safe. Our Rakefile, for whatever reason, needs to download stuff and
then work with it. So we can't use this HTTP library directly -- we need to
wrap synchronization around it.

But, there's still an advantage to running the actual meat of the tasks in
parallel -- maybe we're doing some complex hpricot parsing, and we're
connecting to a potentially-slow server. Ideally, we want to download as fast
as we can, but once it's downloaded, we want to start crunching in worker
threads.

So there's still a benefit to Drake/Multitask, but there's the added
complexity of having to wrap that non-thread-safe library.

Contrived example, I know.

That's an advantage to single-threaded Rake, by the way -- by default, you
don't need to think about any of this. (-j1 isn't an excuse, unless the
Rakefile can force it, because then it's up to the user to figure out what
j-level to use. That should be transparent.)

···

On Tuesday 09 September 2008 17:33:37 . wrote:

33

I'm probably being naive, but couldn't rake catch SIGTERM and handle it the
same way it appears to handle SIGINT?

···

On Thu, Sep 11, 2008 at 11:45:11AM +0900, Jim Weirich wrote:

Jos Backus wrote:
> I'm misremembering. SIGINT seems to work okay, it's SIGTERM that leaves
> orphaned children (with ppid 1) around with rake, presumably because it
> doesn't catch that signal. Same with drake (0.8.1.11.0.1)

What would be a good way to fix this?

--
Jos Backus
jos at catnook.com

34

But I thought we just agreed those two forms should be the same? Now
you are proposing a flag which will make them different.

I think you're missing the larger point, which is that there are still
a million ways to make a mistake with the dependency graph. Your flag
will not come close to saving us.

A tool which analyzes our code and outputs what we meant to write
would be great. But until that glorious technology arrives, tools
will continue to operate on the principle of garbage-in, garbage-out.
As I've unsuccessfully explained in this thread, the problem lies in
our mistakes in thinking.

It seems to me the general solution is to not use -j until you've got
all your ducks in a row.

JL

···

On Sep 9, 7:32 pm, Martin DeMello <martindeme...@gmail.com> wrote:

On Tue, Sep 9, 2008 at 4:13 PM, . <quixoticsycoph...@gmail.com> wrote:

> (and I should hope we all agree), then there is nothing which can save
> us. We are forced to write our Rakefiles correctly. No backwards
> compatibility mode is possible for threads>1.

A --file-order-implies-dependency flag might get us there in a lot of
cases, though of course there's no general solution. Of more value
would be a lint tool that helps convert a rakefile into parallelisable
form.

martin

35

Eric Hodel wrote:

We were able to use drake with only one change due to having working
rakefiles beforehand.

I'm anxious to try it, but as I wrote the gem seems not to work. How
did you do it?

···

--
Posted via http://www.ruby-forum.com/\.

36

There's no such thing as luck in computer programming.

T.

···

On Sep 10, 3:13 pm, Eric Hodel <drbr...@segment7.net> wrote:

On Sep 10, 2008, at 11:06 AM, Trans wrote:

> Is it worth potentially breaking Rakefiles to prevent this sort of
> thing (like drake -j2 or more does)? I'm not so sure. While one might
> consider this Rakefile "bad design" because it doesn't fit the
> original formal notion, it nonetheless does what one would expect it
> to do. I think I'd rather have that, than the potential for ambiguous
> behavior.

They aren't potentially broken, they are broken. If it happens to
work, you've just gotten lucky.

37

> While one might consider this Rakefile "bad design" because it
> doesn't fit the original formal notion, it nonetheless does what one
> would expect it to do. I think I'd rather have that, than the
> potential for ambiguous behavior.

Underspecified dependencies + parallel execution == ambiguous behavior

They are only unspecified according to an interpretation of how things
ought to be. In the current implementation Rake is executing in a
predictable order. One can use it, and people have. Maybe not formally
ideal but the functionality is there. But that's not whast really
concerns me. The issue I was looking at was:

  drake -j2 + Rakefile = ambiguous behavior

So I was suggesting that it would perhaps be better to accept rake's
current implementation behavior; this ambiguity would then not arise;
and instead provide another notation to indicate parallel execution.
My particular idea might not be the best one, I was just looking for a
possible solution that could be useful in itself and address this
issue. Another possibility is just placing a statement at the
beginning of a Rakefile that could be used to indicate that the
rakefile is in fact "j-able".

I thought it prudent to address this b/c, personally, I'd like to see -
j end up in Rake itself. But perhaps it is better to just move forward
and expect people to fix all there old Rakefiles (and lets just hope
nothing really ugly happens when they haven't).

> There's no such thing as luck in computer programming.

Yes, there is.

And his name is _why? :wink: Well, i suppose if we want to take chances,
then there is.

T.

···

On Sep 10, 10:58 pm, "." <quixoticsycoph...@gmail.com> wrote:

On Sep 10, 2:06 pm, Trans <transf...@gmail.com> wrote:

38

They *should* be the same, but if we're discussing legacy rakefiles
where people have implicitly relied on their being different...

I agree that there's really no 'right' thing to do, though - either
you've specified your depgraph properly or you haven't.

martin

···

On Tue, Sep 9, 2008 at 6:13 PM, . <quixoticsycophant@gmail.com> wrote:

On Sep 9, 7:32 pm, Martin DeMello <martindeme...@gmail.com> wrote:

A --file-order-implies-dependency flag might get us there in a lot of
cases, though of course there's no general solution. Of more value
would be a lint tool that helps convert a rakefile into parallelisable
form.

But I thought we just agreed those two forms should be the same? Now
you are proposing a flag which will make them different.

39

I evaluated a prerelease copy. I've not yet tried this release.

···

On Sep 10, 2008, at 12:20 PM, Anton Ivanov wrote:

Eric Hodel wrote:

We were able to use drake with only one change due to having working
rakefiles beforehand.

I'm anxious to try it, but as I wrote the gem seems not to work. How
did you do it?

40

Did you see my response above? What is the output when you run drake
on this

  task :default do
    puts $LOAD_PATH
  end

For some reason rubygems isn't manipulating your $LOAD_PATH correctly,
or something is overriding it.

···

On Sep 10, 3:20 pm, Anton Ivanov <foral...@gmail.com> wrote:

Eric Hodel wrote:
> We were able to use drake with only one change due to having working
> rakefiles beforehand.

I'm anxious to try it, but as I wrote the gem seems not to work. How
did you do it?