DI service change notifications (Syringe)

hi,

i'm playing around with Syringe (http://ruby.jamisbuck.org/syringe/),
and i've got an idea for additional behaviour that i want, though i'm
not sure if its a good idea yet.

what i want, is the option for services to be less static (i.e. not a
singleton where instantiation happens only once. i want more control
over service lifecycle..)

would what i'm asking for be a different "model", to use Syringe
terminology? the ability to flag a service as dirty, and
automatically, all its consumers get flagged as dirty too, for
refreshing on next usage.

contrived pseudocode example:

env = { :log_filename => 'test.log' }
container.register(:log_filename) { env[:log_filename] }
container.register(:logger) { |c| Logger.new(c.log_filename) }

now, since one can change the value in env, when making that change,
the matching :log_filename service would be flagged as dirty, and, on
next usage, the block executed again, and logger logs to different
file without program restart.

good idea? crap?

leon

i am aware that there will have to be a link between env and container
to ensure that the dirty flag will be set, first of all, and set on
the correct service, secondly :slight_smile:

possibly env would be a special service instead of floating Hash, and
access to env would happen through an interceptor gatekeeper...

leon

···

On Sun, 10 Oct 2004 22:34:59 +1300, leon breedt <bitserf@gmail.com> wrote:

contrived pseudocode example:

env = { :log_filename => 'test.log' }
container.register(:log_filename) { env[:log_filename] }
container.register(:logger) { |c| Logger.new(c.log_filename) }

leon breedt wrote:

hi,

i'm playing around with Syringe (http://ruby.jamisbuck.org/syringe/\),

Wow. I'm surprised -- I haven't even made an announcement about it, except on my blog. You realize, I hope, that Syringe is in a HUGE state of flux right now, and the API is guaranteed to change in all kinds of non-backwards-compatible ways, right?

env = { :log_filename => 'test.log' }
container.register(:log_filename) { env[:log_filename] }
container.register(:logger) { |c| Logger.new(c.log_filename) }

now, since one can change the value in env, when making that change,
the matching :log_filename service would be flagged as dirty, and, on
next usage, the block executed again, and logger logs to different
file without program restart.

good idea? crap?

A new service model would not really help in this case, because once the Logger.new constructor is called, the log_filename has already been returned as a string. What you need is to create an object that represents the filename, without _being_ the filename, and have its #to_str method defined to return the filename:

   class LogFilename
     def to_str
       "test.log"
     end
   end

Then, you can register the :log_filename service to an instance of that:

   container.register( :log_filename ) { LogFilename.new }

If you define LogFilename correctly, you could then change the string that gets returned there, and have all consumers of the filename take advantage of that.

But, there's a problem. In your example, you had Logger be the consumer of the filename. Logger is not designed to allow the filename to be changed. You would have to actually have a new logger instance be created and substituted in place, something Ruby doesn't support without using proxy objects (unless you're using evil.rb).

SO. What is needed is a way to allow consumers of consumers of :log_filename (that is to say, consumers of :logger) to detect when the :log_filename service was modified and then grab a new instance of :logger.

What is really needed, though, is an implementation of Logger that allows the filename to be changed on the fly. If that existed, there would be an easier (i.e., more tractable) solution.

···

--
Jamis Buck
jgb3@email.byu.edu
http://www.jamisbuck.org/jamis

leon breedt wrote:

contrived pseudocode example:

env = { :log_filename => 'test.log' }
container.register(:log_filename) { env[:log_filename] }
container.register(:logger) { |c| Logger.new(c.log_filename) }

i am aware that there will have to be a link between env and container
to ensure that the dirty flag will be set, first of all, and set on
the correct service, secondly :slight_smile:

possibly env would be a special service instead of floating Hash, and
access to env would happen through an interceptor gatekeeper...

Arg. After my last rambling message, I realized I might have missed the mark of what you were asking.

Did you want:

   (A) The have existing references to Logger refresh themselves and start logging to the new file? or

   (B) To have existing references to Logger remain unchanged and have new loggers start logging to the new file?

(A) is not so simple to do (as I mentioned in my last post). (B), however, would be much more straightforward.

- Jamis

···

On Sun, 10 Oct 2004 22:34:59 +1300, leon breedt <bitserf@gmail.com> wrote:

--
Jamis Buck
jgb3@email.byu.edu
http://www.jamisbuck.org/jamis

You realize, I hope, that Syringe is in a HUGE state
of flux right now, and the API is guaranteed to change in all kinds of
non-backwards-compatible ways, right?

Understood :slight_smile:

A new service model would not really help in this case, because once the
Logger.new constructor is called, the log_filename has already been
returned as a string.

Refreshing the dependendant services would take care of this
particular problem (more details in my other reply), and wouldn't
require the creation of stub classes for "data" services.

What is really needed, though, is an implementation of Logger that
allows the filename to be changed on the fly. If that existed, there
would be an easier (i.e., more tractable) solution.

The weakness of that is that it requires an API change to Logger,
which would not be required if dependency graph + dirty flag
propagation existed (other post). In addition, you'd get things like
automatic closing (as the previous logger goes out of scope) and
re-opening of the new file for free, without any Logger changes as
well.

However, it might be, that in some cases, service creation is
heavyweight enough that this "push" model is whats required. Or, as in
the example in my other post, if something changes an unrelated value
in the :config store, you don't want all services to be refreshed. In
which case it probably makes sense to have scoped :config* stores for
all configuration pertaining to a particular set of services.

Leon

···

On Sun, 10 Oct 2004 23:15:08 +0900, Jamis Buck <jgb3@email.byu.edu> wrote:

I'm not sure I was clear enough, and I am still of the opinion (A)
would not be *that* complex to implement, bear with me, I'm not that
good at describing it in English :slight_smile:

I wanted the registry to keep track of the dependency graph, so that
when a service was changed, it would be marked dirty. The registry
would then walk the dependency graph in reverse order, and mark all
dependant services as dirty too. Being dirty just means that the next
time a request for a dirty service was made, the creation block
would be executed again, result cached, and marked clean.

I realize "service was changed" is rather vague.

In my case, I would probably write a service to contain configuration
values like "filename" (I don't like mixing data-only services with
services that actually do something). I'd add an domain-specific
interceptor wrapping this configuration service, that would know which
method invocations on the service are 'writes'. The interceptor would
then send a message to the registry, saying "service :config is
dirty". The registry, because it knows the dependencies on :config,
can mark :config and all things that used :config, as dirty.

The registry would know the dependency graph, because: any services
depending on :config would have done a "reg.config.value" in their
construction block.

In effect, this would be (A) above.

Pseudo-code:

reg.register(:config) { Configuration.new }
reg.intercept(:config).with { |reg| ConfigurationInterceptor.new(reg} }
reg.register(:logger) { |reg| Logger.new(reg.config.filename) }

....

somewhere else in application: reg.config.filename = "newfile"

Because the construction block for :logger called reg.config, the
registry is aware that it depends on :config. The reg.config.filename
assign would have been caught by the interceptor as a "write", and
once completed, the interceptor would send a dirty(:config) message to
the registry.

I was thinking, for this particular approach to work, |reg| in the
:logger construction block might need to be a proxy that collects the
services used in the block, and adds them to a dependency graph,
instead of being the registry itself (for thread-safety?)

Hope that gives you more of an idea of what I'm getting at...

Leon

···

On Sun, 10 Oct 2004 23:17:12 +0900, Jamis Buck <jgb3@email.byu.edu> wrote:

Did you want:

   (A) The have existing references to Logger refresh themselves and
start logging to the new file? or

   (B) To have existing references to Logger remain unchanged and have
new loggers start logging to the new file?

(A) is not so simple to do (as I mentioned in my last post). (B),
however, would be much more straightforward.

Goes without saying that for this to work, all classes must use
reg.logger instead of caching a logger reference :slight_smile:

Leon

···

On Mon, 11 Oct 2004 10:36:02 +1300, leon breedt <bitserf@gmail.com> wrote:

In addition, you'd get things like
automatic closing (as the previous logger goes out of scope) and
re-opening of the new file for free, without any Logger changes as
well.

leon breedt wrote:

Pseudo-code:

reg.register(:config) { Configuration.new }
reg.intercept(:config).with { |reg| ConfigurationInterceptor.new(reg} }
reg.register(:logger) { |reg| Logger.new(reg.config.filename) }

....

somewhere else in application: reg.config.filename = "newfile"

Because the construction block for :logger called reg.config, the
registry is aware that it depends on :config. The reg.config.filename
assign would have been caught by the interceptor as a "write", and
once completed, the interceptor would send a dirty(:config) message to
the registry.

Hmmm. But there's still the issue of the reg.config.filename invocation (to use the example above) knowing the context in which it is being called. It would need to know, specifically, which service point (:logger) was being instantiated, so that it could relate the dependency back to that service... hmmm.

I was thinking, for this particular approach to work, |reg| in the
:logger construction block might need to be a proxy that collects the
services used in the block, and adds them to a dependency graph,
instead of being the registry itself (for thread-safety?)

But what about something like this happening:

   reg.register(:logger) { |r| Logger.new( reg.config.filename ) }
   ^^^ ^^^ ^^^

In other words, there is nothing that prevents the block from referring to registries and containers outside the block. This means that it really can't be a proxy object--it has to be the container itself that knows the dependency information.

Perhaps you could use a thread-local variable to keep track of the service that is being constructed. It would need to be a stack, though, since instantiating one service can cascade into multiple service instantiations.

If the #register method stored a stack of service names that are currently being constructed by each thread in a thread-local variable, your interceptor could use that variable to relate the dependency back to the registry...

Hmm. But this falls apart when the container itself is a dependency, since the service that depends thus on the container could query the container at any time, not just during the service's instantiation. This means that the service could effectively have new dependencies at arbitrary points during its lifecycle:

   reg.register(:svc) { SomeNewService.new(reg) }
   reg.svc.reg.another_service
   ...

I guess what I'm feeling is that there is no general way to know, definitively, all of the dependencies of a service, especially in a dynamic language like Ruby. Without putting restrictions on when and how services are defined (like Copland does), I'm not sure how to come up with a general solution to this.

That said, what about a slightly less transparent approach:

   reg.register(:config) { RefreshableProxy.new { Config.new } }
   reg.register(:logger) { RefreshableProxy.new( reg, :config ) {
      Logger.new( reg.config.filename } }

   logger = reg.logger
   reg.config.refresh!
   ...

What the above snippet is trying to demonstrate is a kind of observer pattern. In the first registration, we create a (imaginary) RefreshableProxy instance that wraps the given configuration instance. In the second, we create another RefreshableProxy instance that wraps the Logger.new instantiation. The second one also declares itself to be an observer of the :config service in the given container.

When #refresh! is invoked on a RefreshableProxy, it notifies all of its observers that it changed, and the next time a method is invoked on that proxy it re-executes its associated block. It's really a specialized kind of deferred instantiation.

Does that make sense? It could be added to the framework so that EVERY service is automatically wrapped in a RefreshableProxy, but that would add a lot of overhead that would rarely be needed. Doing it manually requires you to explicitly specify the dependency graph in the form of observers, but also incurs a lot less overhead (in general).

Now, if that is what you were wanting, the above COULD be implemented as a new service model:

   class RefreshableServiceModel
     ...
   end

   reg.service_models[:refreshable] = RefreshableServiceModel
   reg.register( :config, :model=>:refreshable ) { Config.new }
   reg.register( :logger, :model=>:refreshable, :observe=>[:config] ) {
     Logger.new( reg.config.filename ) }

   config = reg.config
   logger = reg.logger
   config.refresh!
   ...

If you would like a concrete implementation of RefreshableServiceModel, let me know. Otherwise, I'll leave it as an exercise for the reader. :wink:

- Jamis

···

--
Jamis Buck
jgb3@email.byu.edu
http://www.jamisbuck.org/jamis

Sounds workable, easier to understand as well. I don't mind having to
explicitly declare the dependencies, easier to maintain as well, no
undercover magic :slight_smile:

The behaviour I'd want is probably just for #refresh! to cause a
#refresh! call on all the observers, but thats just implementation
details.

Thanks for the detailed responses!
Leon

···

On Mon, 11 Oct 2004 09:12:47 +0900, Jamis Buck <jgb3@email.byu.edu> wrote:

Now, if that is what you were wanting, the above COULD be implemented as
a new service model:

I get a feeling you're overengineering here.

Doesn't something like the following work?

class Registry
  def initialize
     @registry = Hash.new
     @active_dependency_registrations =
     @dependencies = Hash.new
  end
  # This is done by method_missing in Syringe
  def get(symbol)
     register_dependency(symbol)
     retval = nil
     begin
        start_dependency(symbol)
        @registery[symbol].call
     finally
        end_dependency(symbol)
     end
  end
  def register(symbol, &block)
     @registery[symbol] = block
  end
  def start_dependency(symbol)
     if (@active_dependencies.include? symbol)
        raise "Circular dependency"
     else
        @active_dependencies.each do |dependent_symbol|
             @dependencies[dependent_symbol] ||=
             @dependencies[dependent_symbol] |= [symbol]
        end
        @active_dependencies |= [symbol]
     end
  end
  def end_dependency(symbol)
       @active_dependencies -= [symbol]
  end
end

It builds a dependency tree only inside the present registry.

It only count the directly called methods from call/yield, but that
should be OK for most cases.

And this is just pseudo-code - it's not the way I'd write it for
production (I'd use Set and Hash.default, and handle threads, and ...)

Eivind.

···

On Mon, 11 Oct 2004 09:12:47 +0900, Jamis Buck <jgb3@email.byu.edu> wrote:

I guess what I'm feeling is that there is no general way to know,
definitively, all of the dependencies of a service, especially in a
dynamic language like Ruby. Without putting restrictions on when and how
services are defined (like Copland does), I'm not sure how to come up
with a general solution to this.

leon breedt wrote:

Now, if that is what you were wanting, the above COULD be implemented as
a new service model:

Sounds workable, easier to understand as well. I don't mind having to
explicitly declare the dependencies, easier to maintain as well, no
undercover magic :slight_smile:

The behaviour I'd want is probably just for #refresh! to cause a
#refresh! call on all the observers, but thats just implementation
details.

I actually thought of that detail after I posted the email, so I agree--that is a feature that ought to exist in the service model as well.

Thanks for the detailed responses!

No problem! I just hope they weren't _too_ detailed. I felt like I was rambling a bit too much... At any rate, I'm glad they were helpful.

- Jamis

···

On Mon, 11 Oct 2004 09:12:47 +0900, Jamis Buck <jgb3@email.byu.edu> wrote:

--
Jamis Buck
jgb3@email.byu.edu
http://www.jamisbuck.org/jamis

Eivind Eklund wrote:

I guess what I'm feeling is that there is no general way to know,
definitively, all of the dependencies of a service, especially in a
dynamic language like Ruby. Without putting restrictions on when and how
services are defined (like Copland does), I'm not sure how to come up
with a general solution to this.

I get a feeling you're overengineering here.

Doesn't something like the following work?

class Registry
  def initialize
     @registry = Hash.new
     @active_dependency_registrations =
     @dependencies = Hash.new
  end
  # This is done by method_missing in Syringe
  def get(symbol)
     register_dependency(symbol)
     retval = nil
     begin
        start_dependency(symbol)
        @registery[symbol].call
     finally
        end_dependency(symbol)
     end
  end
  def register(symbol, &block)
     @registery[symbol] = block
  end
  def start_dependency(symbol)
     if (@active_dependencies.include? symbol)
        raise "Circular dependency"
     else
        @active_dependencies.each do |dependent_symbol|
             @dependencies[dependent_symbol] ||=
             @dependencies[dependent_symbol] |= [symbol]
        end
        @active_dependencies |= [symbol]
     end
  end
  def end_dependency(symbol)
       @active_dependencies -= [symbol]
  end
end

My head hurts. :slight_smile:

It builds a dependency tree only inside the present registry.

Fascinating approach, Eivind. Let me see if I read this right.

* Registering the callback is nothing special -- just storing the callback in the hash.

* When a service is requesed, you register the requested service as a dependency of whatever service is currently being constructed.

It only count the directly called methods from call/yield, but that
should be OK for most cases.

Indeed. However, if this kind of functionality were to be put into the core of the framework, I would personally prefer to see it robust enough to either capture all dependencies, regardless of where or how they are created, or none of them. Given Syringe's current implementation, I don't think an efficient, general-purpose solution meeting that criterion can be devised. I'd love to be proven wrong, though. :slight_smile:

(On a related note: I just registered a RubyForge project under the name "Needle". I think "Needle" is easier to say, spell, and remember, and still captures what "Syringe" was trying to imply, as well. In future posts, I'll be referring to 'Syringe' as 'Needle', since Syringe was just a code-name anyway.)

···

On Mon, 11 Oct 2004 09:12:47 +0900, Jamis Buck <jgb3@email.byu.edu> wrote:

--
Jamis Buck
jgb3@email.byu.edu
http://www.jamisbuck.org/jamis

Eivind Eklund wrote:

[ ... code deleted ... ]

My head hurts. :slight_smile:

Sorry about that.

> It builds a dependency tree only inside the present registry.

Fascinating approach, Eivind. Let me see if I read this right.

* Registering the callback is nothing special -- just storing the
callback in the hash.

* When a service is requesed, you register the requested service as a
dependency of whatever service is currently being constructed.

Exactly. And only inside the present registry.

> It only count the directly called methods from call/yield, but that
> should be OK for most cases.

Indeed. However, if this kind of functionality were to be put into the
core of the framework, I would personally prefer to see it robust enough
to either capture all dependencies, regardless of where or how they are
created, or none of them. Given Syringe's current implementation, I
don't think an efficient, general-purpose solution meeting that
criterion can be devised. I'd love to be proven wrong, though. :slight_smile:

I don't think any can be devised, period. And thus, I think that the
design criterion is inappropriate - if a problem is too hard, find a
smaller and better problem to solve :slight_smile:

In this case, I think that being able to register the dependency tree
for the cases where somebody declares all of their construction "in
the same place" might be useful. I also guess this covers at least
95% of the relevant cases, and that the cases where the holes are made
will be obvious - and that they won't be obvious if we try to engineer
a "complete" solution. And it is easy to create something that lets
us add an extra manual dependency for those cases.

Having to manually track the dependency trees separately from the
instation heavily violates the Don't Repeat Yourself principle, so if
there is any chance of this being used for complex cases, I think that
a capability to auto-record is important.

Then again, I've not used any of this in practice, so I may be totally
off the chart.

Eivind.

···

On Tue, 12 Oct 2004 00:48:05 +0900, Jamis Buck <jgb3@email.byu.edu> wrote:

Eivind Eklund wrote:

Indeed. However, if this kind of functionality were to be put into the
core of the framework, I would personally prefer to see it robust enough
to either capture all dependencies, regardless of where or how they are
created, or none of them. Given Syringe's current implementation, I
don't think an efficient, general-purpose solution meeting that
criterion can be devised. I'd love to be proven wrong, though. :slight_smile:

I don't think any can be devised, period. And thus, I think that the
design criterion is inappropriate - if a problem is too hard, find a
smaller and better problem to solve :slight_smile:

True, but only if it's a problem worth solving. :slight_smile: In my experience with DI, I've never needed to query the dependency graph. That doesn't mean it isn't a useful feature--I'm just saying it isn't something that would get used very often.

That said, is it worth spending a ton of effort on? According to the 80-20 rule--I say not. Does the currently implementation require a violation of DRY, as you pointed out, to accomplish dependency graph traversal? Yes. But given how often that feature is needed, I think it's a fair trade.

If, down the road, my assumptions are proven false, I'll be happy to rip the internals apart and rebuild them to support this feature. But I've got more immediately useful features to solidify first. :slight_smile: This whole needle framework is very experimental at this stage. If someone else wants to hack on the sources and add dependency graph traversal, feel free. Needle will be in CVS on RubyForge (pending approval) within the next few days. I will be glad of any submitted patches.

In this case, I think that being able to register the dependency tree
for the cases where somebody declares all of their construction "in
the same place" might be useful. I also guess this covers at least
95% of the relevant cases, and that the cases where the holes are made
will be obvious - and that they won't be obvious if we try to engineer
a "complete" solution. And it is easy to create something that lets
us add an extra manual dependency for those cases.

True, but the holes, IMO, introduce an element of surprise. If dependencies are tracked under 95% of the cases, and you don't understand how the system is really working, you would end up with bugs in your code when you declare a dependency differently from the expected process.

I'd really rather users of needle not have to understand how needle works its magic.

Having to manually track the dependency trees separately from the
instation heavily violates the Don't Repeat Yourself principle, so if
there is any chance of this being used for complex cases, I think that
a capability to auto-record is important.

Then again, I've not used any of this in practice, so I may be totally
off the chart.

Me, too. :slight_smile: The only use I've seen for this is the one that Leon posted with being able to refresh a service's implementation and have all services that are dependant upon the refreshed service automatically refresh as well. How often is _that_ needed? I don't know. _I've_ never needed it, but Leon has. No one else has piped up yet to say whether _they've_ needed it, so it's still unclear how generally useful such a feature would be.

So, given that we're both kind of shouting into a vacuum here, we might as well just hold off on this until more real-life use cases pop up. :slight_smile: I'm expecting many different iterations and rewrites of Needle before it is really "mature", since this is an approach to DI that's never been taken before.

Thanks, Eivind, for your feedback! It's great food for thought, and it challenges my assumptions, which is always a good thing.

···

On Tue, 12 Oct 2004 00:48:05 +0900, Jamis Buck <jgb3@email.byu.edu> wrote:

--
Jamis Buck
jgb3@email.byu.edu
http://www.jamisbuck.org/jamis