# Thinking Aloud: Refactoring from nested loops to enumerators and lambda's

So I'm just thinking out aloud here.

Being an Old School Programmer I tend to naturally to write ever more
deeply nested loops.

I hate myself when I do this because it’s hard to test especially if
some of the loops have nasty external side effects, it’s hard to
reuse, it’s hard to refactor.

So I’m trying two new patterns....

* Passing enumerators as parameters so I can pull out inner loops as
standalone functions without calling the entire function every time..

* Creating functions that return lambdas, so I can pass an inner loop
in as a parameter.

Walk with me this is going to be long…. the example is a teaching
example / dojo exercise for myself, so excuse me it it slightly
contrived.

(In the following, a line of ===== indicates the next, slightly
different version of the code)

···

==========================================

Here is a typical chunk of my code…

def nested( a, b, c)

stuff_a = func_a(a)
stuff_b = func_b(b)
stuff_c = func_c(c)
result = {}
func_1( stuff_a) do |a1|
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|

result[a3] = func_f( a3)
end
end
end

result
end

It’s fairly clear but has a few gotchas.

* It’s unclear which part of the code actually depends on which parameters.
* In this toy example, the function is small… but a Real Life nested
loop function like this can quickly grow hideously large.
* As a “premature optimization” I have factored out subexpressions
that do not alter within the loops, resulting in large scopes for
variables that are only used inside the loops.
* func_1(), func_2(), func_3() yield a stream of things….but a stream
of things should just be an enumerable!

==========================================

Ok, so try 2… reduce the scope of the stuff_* variables, a pessimation..

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|

result[a3] = func_f( a3)
end
end
end

result
end

==========================================

I can extract the inner loop as a function, but my parameter list balloons…

def inner_2( a2, c, result)
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
inner_2( a2, c, result)
end
end

result
end

and I still reevaluate func_c for every loop!

==========================================
If I use a closure instead, my parameter list collapses again…

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( a2){
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
}
func_2( stuff_d) do |a2|
inner_2.call( a2)
end
end

result
end

==========================================
But I still have a pessimization, so if I could pass an enumerator
around…. So lets try convert func_2 to an enumerator….

def func_2( j)
.....lots of code and a...
yield a2
...lots more code
end

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( a2){
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
}
func_2( stuff_d).each do |a2|
inner_2.call( a2)
end
end

result
end

==========================================

Then pass the enumerator in, and then we can stop the silly
re-evaluation of func_c on every loop…

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( e){
stuff_c = func_c(c)

e.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end
}

inner_2.call( func_2( stuff_d))
end

result
end

==========================================

And I can keep going with func_1….

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
inner_1 = ->( e1, inner_2){
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d))
end
}

inner_2 = ->( e2){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end
}

inner_1.call( func_1( stuff_a), inner_2)

result
end

==========================================

I can reduce scope of result and move it to the outermost level….

def nested( a, b, c)

stuff_a = func_a(a)

inner_1 = ->( e1, inner_2, &block){
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
}

inner_2 = ->( e2,&block){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}

result = {}
inner_1.call( func_1( stuff_a), inner_2) do |a3|
result[a3] = func_f( a3)
end
result
end

==========================================

I can convert the inner_1 lambda to a vanilla method and convert that
to an Enumerator…

def inner_1( b, e1, inner_2, &block)
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
end

def nested( a, b, c)

stuff_a = func_a(a)

inner_2 = ->( e2,&block){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}

result = {}
inner_1( b, func_1( stuff_a), inner_2).each do |a3|
result[a3] = func_f( a3)
end
result
end

==========================================

Since inner_1 is just a vanilla enum, I can use each_with_object

I can also extract inner_2 as a method that returns a lambda…

def inner_1( b, e1, inner_2, &block)
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
end

def inner_2( c)
stuff_c = func_c(c)

->( e2,&block){

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}
end

def nested( a, b, c)

stuff_a = func_a(a)

inner_1( b, func_1( stuff_a), inner_2( c)).each_with_object({}) do

a3, result|

result[a3] = func_f( a3)
end
end

Note func_c is now only evaluated once again, It’s testable it, it’s
reusable in the vanilla “everything is just Enumerable” sense.

As I said, I'm just thinking aloud trying to fix a common anti-pattern
in my own code.

--
John Carter
Tait Electronics
PO Box 1645 Christchurch
New Zealand

--
This Communication is Confidential. We only send and receive email on the

basis of the terms set out at www.taitradio.com/email_disclaimer

Do you have a real world example of a code that needs this pattern? The
problem with generalized or anonymized examples is they lack the context
that would make solving the problem clearer.

I would suggest making a minimal example that is tested that you can share
with valid input and what output you expect it to return. I suspect that
you're conflating a lot of ideas here but can't really understand the code
from the examples.

I'll have to reread the code a few times to understand what you're doing
here, but my initial impression is to break it apart into simple-use
functions that do one and only one thing, and compose those in turn.

···

On Sun, Jun 30, 2019 at 11:13 PM John Carter <john.carter@taitradio.com> wrote:

So I'm just thinking out aloud here.

Being an Old School Programmer I tend to naturally to write ever more
deeply nested loops.

I hate myself when I do this because it’s hard to test especially if
some of the loops have nasty external side effects, it’s hard to
reuse, it’s hard to refactor.

So I’m trying two new patterns....

* Passing enumerators as parameters so I can pull out inner loops as
standalone functions without calling the entire function every time..

* Creating functions that return lambdas, so I can pass an inner loop
in as a parameter.

Walk with me this is going to be long…. the example is a teaching
example / dojo exercise for myself, so excuse me it it slightly
contrived.

(In the following, a line of ===== indicates the next, slightly
different version of the code)

==========================================

Here is a typical chunk of my code…

def nested( a, b, c)

stuff_a = func_a(a)
stuff_b = func_b(b)
stuff_c = func_c(c)
result = {}
func_1( stuff_a) do |a1|
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|

result[a3] = func_f( a3)
end
end
end

result
end

It’s fairly clear but has a few gotchas.

* It’s unclear which part of the code actually depends on which parameters.
* In this toy example, the function is small… but a Real Life nested
loop function like this can quickly grow hideously large.
* As a “premature optimization” I have factored out subexpressions
that do not alter within the loops, resulting in large scopes for
variables that are only used inside the loops.
* func_1(), func_2(), func_3() yield a stream of things….but a stream
of things should just be an enumerable!

==========================================

Ok, so try 2… reduce the scope of the stuff_* variables, a pessimation..

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|

result[a3] = func_f( a3)
end
end
end

result
end

==========================================

I can extract the inner loop as a function, but my parameter list balloons…

def inner_2( a2, c, result)
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
inner_2( a2, c, result)
end
end

result
end

and I still reevaluate func_c for every loop!

==========================================
If I use a closure instead, my parameter list collapses again…

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( a2){
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
}
func_2( stuff_d) do |a2|
inner_2.call( a2)
end
end

result
end

==========================================
But I still have a pessimization, so if I could pass an enumerator
around…. So lets try convert func_2 to an enumerator….

def func_2( j)
.....lots of code and a...
yield a2
...lots more code
end

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( a2){
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
}
func_2( stuff_d).each do |a2|
inner_2.call( a2)
end
end

result
end

==========================================

Then pass the enumerator in, and then we can stop the silly
re-evaluation of func_c on every loop…

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( e){
stuff_c = func_c(c)

e.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end
}

inner_2.call( func_2( stuff_d))
end

result
end

==========================================

And I can keep going with func_1….

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
inner_1 = ->( e1, inner_2){
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d))
end
}

inner_2 = ->( e2){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end
}

inner_1.call( func_1( stuff_a), inner_2)

result
end

==========================================

I can reduce scope of result and move it to the outermost level….

def nested( a, b, c)

stuff_a = func_a(a)

inner_1 = ->( e1, inner_2, &block){
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
}

inner_2 = ->( e2,&block){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}

result = {}
inner_1.call( func_1( stuff_a), inner_2) do |a3|
result[a3] = func_f( a3)
end
result
end

==========================================

I can convert the inner_1 lambda to a vanilla method and convert that
to an Enumerator…

def inner_1( b, e1, inner_2, &block)
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
end

def nested( a, b, c)

stuff_a = func_a(a)

inner_2 = ->( e2,&block){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}

result = {}
inner_1( b, func_1( stuff_a), inner_2).each do |a3|
result[a3] = func_f( a3)
end
result
end

==========================================

Since inner_1 is just a vanilla enum, I can use each_with_object

I can also extract inner_2 as a method that returns a lambda…

def inner_1( b, e1, inner_2, &block)
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
end

def inner_2( c)
stuff_c = func_c(c)

->( e2,&block){

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}
end

def nested( a, b, c)

stuff_a = func_a(a)

inner_1( b, func_1( stuff_a), inner_2( c)).each_with_object({}) do
>a3, result|
result[a3] = func_f( a3)
end
end

Note func_c is now only evaluated once again, It’s testable it, it’s
reusable in the vanilla “everything is just Enumerable” sense.

As I said, I'm just thinking aloud trying to fix a common anti-pattern
in my own code.

--
John Carter
Tait Electronics
PO Box 1645 Christchurch
New Zealand

--
This Communication is Confidential. We only send and receive email on the

basis of the terms set out at www.taitradio.com/email_disclaimer

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>

Most well written libraries tend not to have this
anti-pattern...because it inhibits re-use.

I designed that example to have all the worst features of that class
of problem... ie. the dependencies on the parameters scattered across
the whole function, and coupling between values computed in outer
loops being used in the inner loops with some values factored out of
the loops.

but my initial impression is to break it apart into simple-use functions that do one and only one thing, and compose those in turn.

The gotcha with that is unless you're careful, the inner functions end
up recomputing values on every iteration, or you pass in a largish
collection of parameters which are really tightly coupled to the inner
function and are entirely uncoupled to the outer functions.

ie. You cannot reuse the inner functions because they are tightly
coupled to the outer.

···

On Mon, Jul 1, 2019 at 4:30 PM Brandon Weaver <keystonelemur@gmail.com> wrote:

Do you have a real world example of a code that needs this pattern? The problem with generalized or anonymized examples is they lack the context that would make solving the problem clearer.

I would suggest making a minimal example that is tested that you can share with valid input and what output you expect it to return. I suspect that you're conflating a lot of ideas here but can't really understand the code from the examples.

I'll have to reread the code a few times to understand what you're doing here, but my initial impression is to break it apart into simple-use functions that do one and only one thing, and compose those in turn.

On Sun, Jun 30, 2019 at 11:13 PM John Carter <john.carter@taitradio.com> wrote:

So I'm just thinking out aloud here.

Being an Old School Programmer I tend to naturally to write ever more
deeply nested loops.

I hate myself when I do this because it’s hard to test especially if
some of the loops have nasty external side effects, it’s hard to
reuse, it’s hard to refactor.

So I’m trying two new patterns....

* Passing enumerators as parameters so I can pull out inner loops as
standalone functions without calling the entire function every time..

* Creating functions that return lambdas, so I can pass an inner loop
in as a parameter.

Walk with me this is going to be long…. the example is a teaching
example / dojo exercise for myself, so excuse me it it slightly
contrived.

(In the following, a line of ===== indicates the next, slightly
different version of the code)

==========================================

Here is a typical chunk of my code…

def nested( a, b, c)

stuff_a = func_a(a)
stuff_b = func_b(b)
stuff_c = func_c(c)
result = {}
func_1( stuff_a) do |a1|
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|

result[a3] = func_f( a3)
end
end
end

result
end

It’s fairly clear but has a few gotchas.

* It’s unclear which part of the code actually depends on which parameters.
* In this toy example, the function is small… but a Real Life nested
loop function like this can quickly grow hideously large.
* As a “premature optimization” I have factored out subexpressions
that do not alter within the loops, resulting in large scopes for
variables that are only used inside the loops.
* func_1(), func_2(), func_3() yield a stream of things….but a stream
of things should just be an enumerable!

==========================================

Ok, so try 2… reduce the scope of the stuff_* variables, a pessimation..

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|

result[a3] = func_f( a3)
end
end
end

result
end

==========================================

I can extract the inner loop as a function, but my parameter list balloons…

def inner_2( a2, c, result)
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
inner_2( a2, c, result)
end
end

result
end

and I still reevaluate func_c for every loop!

==========================================
If I use a closure instead, my parameter list collapses again…

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( a2){
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
}
func_2( stuff_d) do |a2|
inner_2.call( a2)
end
end

result
end

==========================================
But I still have a pessimization, so if I could pass an enumerator
around…. So lets try convert func_2 to an enumerator….

def func_2( j)
.....lots of code and a...
yield a2
...lots more code
end

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( a2){
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
}
func_2( stuff_d).each do |a2|
inner_2.call( a2)
end
end

result
end

==========================================

Then pass the enumerator in, and then we can stop the silly
re-evaluation of func_c on every loop…

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( e){
stuff_c = func_c(c)

e.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end
}

inner_2.call( func_2( stuff_d))
end

result
end

==========================================

And I can keep going with func_1….

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
inner_1 = ->( e1, inner_2){
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d))
end
}

inner_2 = ->( e2){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end
}

inner_1.call( func_1( stuff_a), inner_2)

result
end

==========================================

I can reduce scope of result and move it to the outermost level….

def nested( a, b, c)

stuff_a = func_a(a)

inner_1 = ->( e1, inner_2, &block){
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
}

inner_2 = ->( e2,&block){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}

result = {}
inner_1.call( func_1( stuff_a), inner_2) do |a3|
result[a3] = func_f( a3)
end
result
end

==========================================

I can convert the inner_1 lambda to a vanilla method and convert that
to an Enumerator…

def inner_1( b, e1, inner_2, &block)
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
end

def nested( a, b, c)

stuff_a = func_a(a)

inner_2 = ->( e2,&block){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}

result = {}
inner_1( b, func_1( stuff_a), inner_2).each do |a3|
result[a3] = func_f( a3)
end
result
end

==========================================

Since inner_1 is just a vanilla enum, I can use each_with_object

I can also extract inner_2 as a method that returns a lambda…

def inner_1( b, e1, inner_2, &block)
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
end

def inner_2( c)
stuff_c = func_c(c)

->( e2,&block){

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}
end

def nested( a, b, c)

stuff_a = func_a(a)

inner_1( b, func_1( stuff_a), inner_2( c)).each_with_object({}) do
>a3, result|
result[a3] = func_f( a3)
end
end

Note func_c is now only evaluated once again, It’s testable it, it’s
reusable in the vanilla “everything is just Enumerable” sense.

As I said, I'm just thinking aloud trying to fix a common anti-pattern
in my own code.

--
John Carter
Tait Electronics
PO Box 1645 Christchurch
New Zealand

--
This Communication is Confidential. We only send and receive email on the

basis of the terms set out at www.taitradio.com/email_disclaimer

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>

--
John Carter
Phone : (64)(3) 358 6639
Tait Electronics
PO Box 1645 Christchurch
New Zealand

--
This Communication is Confidential. We only send and receive email on the

basis of the terms set out at www.taitradio.com/email_disclaimer

I tend to think of loops -- at least loops with blocks -- as a code smell.

So the first step for me is to make the body of each of those loops a function. After that, if you are finding that there are too many parameters being passed around, then that might be an indication that you the design is not optimal (only _might_). For example, perhaps you could use a helper class that carries some of those parameters as persistent state.

Then, too, if your loops have external side effects, that's another line of attack. I kind of want each of my functions to have an external effect, or return a value, but definitely not both.

Of course

``````Result = []
func_x( stuff) do |a|
result << func_y( a)
end
result
``````

Is really `func_x(stuff).map(&:func_y)`, so there's that, too.

···

-----Original Message-----
From: ruby-talk [mailto:ruby-talk-bounces@ruby-lang.org] On Behalf Of John Carter
Sent: 01 July 2019 05:13
To: Ruby users
Subject: Thinking Aloud: Refactoring from nested loops to enumerators and lambda's

So I'm just thinking out aloud here.

Being an Old School Programmer I tend to naturally to write ever more
deeply nested loops.

I hate myself when I do this because it’s hard to test especially if
some of the loops have nasty external side effects, it’s hard to
reuse, it’s hard to refactor.

So I’m trying two new patterns....

* Passing enumerators as parameters so I can pull out inner loops as
standalone functions without calling the entire function every time..

* Creating functions that return lambdas, so I can pass an inner loop
in as a parameter.

Walk with me this is going to be long…. the example is a teaching
example / dojo exercise for myself, so excuse me it it slightly
contrived.

(In the following, a line of ===== indicates the next, slightly
different version of the code)

==========================================

Here is a typical chunk of my code…

def nested( a, b, c)

stuff_a = func_a(a)
stuff_b = func_b(b)
stuff_c = func_c(c)
result = {}
func_1( stuff_a) do |a1|
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|

result[a3] = func_f( a3)
end
end
end

result
end

It’s fairly clear but has a few gotchas.

* It’s unclear which part of the code actually depends on which parameters.
* In this toy example, the function is small… but a Real Life nested
loop function like this can quickly grow hideously large.
* As a “premature optimization” I have factored out subexpressions
that do not alter within the loops, resulting in large scopes for
variables that are only used inside the loops.
* func_1(), func_2(), func_3() yield a stream of things….but a stream
of things should just be an enumerable!

==========================================

Ok, so try 2… reduce the scope of the stuff_* variables, a pessimation..

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|

result[a3] = func_f( a3)
end
end
end

result
end

==========================================

I can extract the inner loop as a function, but my parameter list balloons…

def inner_2( a2, c, result)
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
inner_2( a2, c, result)
end
end

result
end

and I still reevaluate func_c for every loop!

==========================================
If I use a closure instead, my parameter list collapses again…

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( a2){
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
}
func_2( stuff_d) do |a2|
inner_2.call( a2)
end
end

result
end

==========================================
But I still have a pessimization, so if I could pass an enumerator
around…. So lets try convert func_2 to an enumerator….

def func_2( j)
.....lots of code and a...
yield a2
...lots more code
end

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( a2){
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
}
func_2( stuff_d).each do |a2|
inner_2.call( a2)
end
end

result
end

==========================================

Then pass the enumerator in, and then we can stop the silly
re-evaluation of func_c on every loop…

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( e){
stuff_c = func_c(c)

e.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end
}

inner_2.call( func_2( stuff_d))
end

result
end

==========================================

And I can keep going with func_1….

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
inner_1 = ->( e1, inner_2){
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d))
end
}

inner_2 = ->( e2){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end
}

inner_1.call( func_1( stuff_a), inner_2)

result
end

==========================================

I can reduce scope of result and move it to the outermost level….

def nested( a, b, c)

stuff_a = func_a(a)

inner_1 = ->( e1, inner_2, &block){
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
}

inner_2 = ->( e2,&block){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}

result = {}
inner_1.call( func_1( stuff_a), inner_2) do |a3|
result[a3] = func_f( a3)
end
result
end

==========================================

I can convert the inner_1 lambda to a vanilla method and convert that
to an Enumerator…

def inner_1( b, e1, inner_2, &block)
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
end

def nested( a, b, c)

stuff_a = func_a(a)

inner_2 = ->( e2,&block){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}

result = {}
inner_1( b, func_1( stuff_a), inner_2).each do |a3|
result[a3] = func_f( a3)
end
result
end

==========================================

Since inner_1 is just a vanilla enum, I can use each_with_object

I can also extract inner_2 as a method that returns a lambda…

def inner_1( b, e1, inner_2, &block)
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
end

def inner_2( c)
stuff_c = func_c(c)

->( e2,&block){

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}
end

def nested( a, b, c)

stuff_a = func_a(a)

inner_1( b, func_1( stuff_a), inner_2( c)).each_with_object({}) do

a3, result|

result[a3] = func_f( a3)
end
end

Note func_c is now only evaluated once again, It’s testable it, it’s
reusable in the vanilla “everything is just Enumerable” sense.

As I said, I'm just thinking aloud trying to fix a common anti-pattern
in my own code.

--
John Carter
Tait Electronics
PO Box 1645 Christchurch
New Zealand

--
This Communication is Confidential. We only send and receive email on the

basis of the terms set out at www.taitradio.com/email_disclaimer

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>

Please note that we have updated our privacy policy in line with new data protection regulations. Please refer to our website to view the ways in which we handle your data.

For example, perhaps you could use a helper class that carries some of those parameters as persistent state.

Yup. And if you think about it, a nameless helper class is to preserve
local state is just a closure.

···

On Mon, Jul 1, 2019 at 7:44 PM Andy Jones <Andy.Jones@jameshall.co.uk> wrote:

I tend to think of loops -- at least loops with blocks -- as a code smell.

So the first step for me is to make the body of each of those loops a function. After that, if you are finding that there are too many parameters being passed around, then that might be an indication that you the design is not optimal (only _might_). For example, perhaps you could use a helper class that carries some of those parameters as persistent state.

Then, too, if your loops have external side effects, that's another line of attack. I kind of want each of my functions to have an external effect, or return a value, but definitely not both.

Of course

``````Result = []
func_x( stuff) do |a|
result << func_y( a)
end
result
``````

Is really `func_x(stuff).map(&:func_y)`, so there's that, too.

-----Original Message-----
From: ruby-talk [mailto:ruby-talk-bounces@ruby-lang.org] On Behalf Of John Carter
Sent: 01 July 2019 05:13
To: Ruby users
Subject: Thinking Aloud: Refactoring from nested loops to enumerators and lambda's

So I'm just thinking out aloud here.

Being an Old School Programmer I tend to naturally to write ever more
deeply nested loops.

I hate myself when I do this because it’s hard to test especially if
some of the loops have nasty external side effects, it’s hard to
reuse, it’s hard to refactor.

So I’m trying two new patterns....

* Passing enumerators as parameters so I can pull out inner loops as
standalone functions without calling the entire function every time..

* Creating functions that return lambdas, so I can pass an inner loop
in as a parameter.

Walk with me this is going to be long…. the example is a teaching
example / dojo exercise for myself, so excuse me it it slightly
contrived.

(In the following, a line of ===== indicates the next, slightly
different version of the code)

==========================================

Here is a typical chunk of my code…

def nested( a, b, c)

stuff_a = func_a(a)
stuff_b = func_b(b)
stuff_c = func_c(c)
result = {}
func_1( stuff_a) do |a1|
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|

result[a3] = func_f( a3)
end
end
end

result
end

It’s fairly clear but has a few gotchas.

* It’s unclear which part of the code actually depends on which parameters.
* In this toy example, the function is small… but a Real Life nested
loop function like this can quickly grow hideously large.
* As a “premature optimization” I have factored out subexpressions
that do not alter within the loops, resulting in large scopes for
variables that are only used inside the loops.
* func_1(), func_2(), func_3() yield a stream of things….but a stream
of things should just be an enumerable!

==========================================

Ok, so try 2… reduce the scope of the stuff_* variables, a pessimation..

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|

result[a3] = func_f( a3)
end
end
end

result
end

==========================================

I can extract the inner loop as a function, but my parameter list balloons…

def inner_2( a2, c, result)
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
inner_2( a2, c, result)
end
end

result
end

and I still reevaluate func_c for every loop!

==========================================
If I use a closure instead, my parameter list collapses again…

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( a2){
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
}
func_2( stuff_d) do |a2|
inner_2.call( a2)
end
end

result
end

==========================================
But I still have a pessimization, so if I could pass an enumerator
around…. So lets try convert func_2 to an enumerator….

def func_2( j)
.....lots of code and a...
yield a2
...lots more code
end

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( a2){
stuff_c = func_c(c)
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
}
func_2( stuff_d).each do |a2|
inner_2.call( a2)
end
end

result
end

==========================================

Then pass the enumerator in, and then we can stop the silly
re-evaluation of func_c on every loop…

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
func_1( stuff_a) do |a1|
stuff_b = func_b(b)
stuff_d = func_d( a1 + stuff_b)
inner_2 = ->( e){
stuff_c = func_c(c)

e.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end
}

inner_2.call( func_2( stuff_d))
end

result
end

==========================================

And I can keep going with func_1….

def nested( a, b, c)

result = {}
stuff_a = func_a(a)
inner_1 = ->( e1, inner_2){
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d))
end
}

inner_2 = ->( e2){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|
result[a3] = func_f( a3)
end
end
}

inner_1.call( func_1( stuff_a), inner_2)

result
end

==========================================

I can reduce scope of result and move it to the outermost level….

def nested( a, b, c)

stuff_a = func_a(a)

inner_1 = ->( e1, inner_2, &block){
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
}

inner_2 = ->( e2,&block){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}

result = {}
inner_1.call( func_1( stuff_a), inner_2) do |a3|
result[a3] = func_f( a3)
end
result
end

==========================================

I can convert the inner_1 lambda to a vanilla method and convert that
to an Enumerator…

def inner_1( b, e1, inner_2, &block)
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
end

def nested( a, b, c)

stuff_a = func_a(a)

inner_2 = ->( e2,&block){
stuff_c = func_c(c)

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}

result = {}
inner_1( b, func_1( stuff_a), inner_2).each do |a3|
result[a3] = func_f( a3)
end
result
end

==========================================

Since inner_1 is just a vanilla enum, I can use each_with_object

I can also extract inner_2 as a method that returns a lambda…

def inner_1( b, e1, inner_2, &block)
stuff_b = func_b(b)
e1.each do |a1|
stuff_d = func_d( a1 + stuff_b)

inner_2.call( func_2( stuff_d), &block)
end
end

def inner_2( c)
stuff_c = func_c(c)

->( e2,&block){

e2.each do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e,&block)
end
}
end

def nested( a, b, c)

stuff_a = func_a(a)

inner_1( b, func_1( stuff_a), inner_2( c)).each_with_object({}) do
>a3, result|
result[a3] = func_f( a3)
end
end

Note func_c is now only evaluated once again, It’s testable it, it’s
reusable in the vanilla “everything is just Enumerable” sense.

As I said, I'm just thinking aloud trying to fix a common anti-pattern
in my own code.

--
John Carter
Tait Electronics
PO Box 1645 Christchurch
New Zealand

--
This Communication is Confidential. We only send and receive email on the

basis of the terms set out at www.taitradio.com/email_disclaimer

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>

Please note that we have updated our privacy policy in line with new data protection regulations. Please refer to our website to view the ways in which we handle your data.

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>

--
John Carter
Phone : (64)(3) 358 6639
Tait Electronics
PO Box 1645 Christchurch
New Zealand

--
This Communication is Confidential. We only send and receive email on the

basis of the terms set out at www.taitradio.com/email_disclaimer

I tend to think of loops -- at least loops with blocks -- as a code smell.

Why?

So the first step for me is to make the body of each of those loops a
function.

Do you mean this?

foo.each do |item|
puts item
# more
end

becomes

def nest(item)
puts item
# more
end

...

foo.each(&:nest)

or - more conventionally

foo.each {|item| nest(item)}

I would not do that as a general rule because now you distribute the logic
in two places which makes it more cumbersome to read. If the same logic is
used in different places then this might make sense (reuse).

Fun nitpick: technically a block is a function - albeit an anonymous one.

After that, if you are finding that there are too many parameters being
passed around, then that might be an indication that you the design is not
optimal (only _might_). For example, perhaps you could use a helper class
that carries some of those parameters as persistent state.

I agree, it might indicate an issue how state and logic are distributed
across classes and methods.

Then, too, if your loops have external side effects, that's another line
of attack. I kind of want each of my functions to have an external effect,
or return a value, but definitely not both.

Sounds good - but I want to mull a bit more about this. The issue I have is
more a formal one: I have come to learn that strict rules are easily
phrased but usually sooner or later you hit an aspect of reality where you
either have to weaken the rule (e.g. allow exceptions) or bend to code in
"amazing" ways to adhere to the strict rule. Reality is so full of
surprises and we all love a simple rule - only more often than not they are
not compatible in my experience.

Of course

``````Result = []
func_x( stuff) do |a|
result << func_y( a)
end
result
``````

Is really `func_x(stuff).map(&:func_y)`, so there's that, too.

Another good point. #inject could also be used here.

Kind regards

robert

···

On Mon, Jul 1, 2019 at 9:44 AM Andy Jones <Andy.Jones@jameshall.co.uk> wrote:

--
[guy, jim, charlie].each {|him| remember.him do |as, often| as.you_can -
without end}
http://blog.rubybestpractices.com/

So I'm just thinking out aloud here.

Being an Old School Programmer I tend to naturally to write ever more
deeply nested loops.

I hate myself when I do this because it’s hard to test especially if
some of the loops have nasty external side effects, it’s hard to
reuse, it’s hard to refactor.

So I’m trying two new patterns....

* Passing enumerators as parameters so I can pull out inner loops as
standalone functions without calling the entire function every time..

* Creating functions that return lambdas, so I can pass an inner loop
in as a parameter.

Walk with me this is going to be long…. the example is a teaching
example / dojo exercise for myself, so excuse me it it slightly
contrived.

(In the following, a line of ===== indicates the next, slightly
different version of the code)

==========================================

Here is a typical chunk of my code…

def nested( a, b, c)

stuff_a = func_a(a)
stuff_b = func_b(b)
stuff_c = func_c(c)
result = {}
func_1( stuff_a) do |a1|
stuff_d = func_d( a1 + stuff_b)
func_2( stuff_d) do |a2|
stuff_e = func_e( a2 + stuff_c)

func_3( stuff_e) do |a3|

result[a3] = func_f( a3)
end
end
end

result
end

thoughts:
* The pattern implies that func_1, func_2 and func_3 do an iteration
internally. Why not extract the Enumerable directly and use that to make
the iteration more visible? Alternatively, if these function filter you
could write a getter that returns an Enumerator.
* Why not just put the logic of the iteration and body in classes of
stuff_a, stuff_d and stuff_e?
* func_d and func_e might be better off having two arguments and doing
* I think I really agree to Brandon: without a more real example and
without knowledge what all the methods actually do it is extremely
difficult to come up with good suggestions.

It’s fairly clear but has a few gotchas.

* It’s unclear which part of the code actually depends on which parameters.
* In this toy example, the function is small… but a Real Life nested
loop function like this can quickly grow hideously large.

But that is an issue in itself that points to issues with the design of the
data structures or logic.

* As a “premature optimization” I have factored out subexpressions
that do not alter within the loops, resulting in large scopes for
variables that are only used inside the loops.

The method in this example is pretty short so I do not see an issue with
that. In other cases that problem automatically goes away if you distribute
the logic properly across multiple methods / functions.

* func_1(), func_2(), func_3() yield a stream of things….but a stream
of things should just be an enumerable!

Exactly, see above.

Kind regards

robert

···

On Mon, Jul 1, 2019 at 6:13 AM John Carter <john.carter@taitradio.com> wrote: