rex goxman wrote in post #1067433:
Well, for the purposes here, if Ruby's threads were green threads, yes
they would have been comparable to Erlang green processes. My emphasis
is far more on the "green" and far less on whether it is a process or a
thread.
Ruby 1.8 *has* green threads. It's explained in Xavier's link, and I
also mentioned it above.
You seem to misread "some implementations have native threads" as "all
have native threads".
--
Posted via http://www.ruby-forum.com/\.
Fibers are pinned to the threads they are created in and cannot be resumed
in other threads
On Thu, Jul 5, 2012 at 9:45 AM, Florian Gilcher <flo@andersground.net>wrote:
Couldn't a similar design be achieved by using Fibers and a Thread pool
that grabs work from a list of ready Fibers?
--
Tony Arcieri
I assure you, I am quite familiar with Erlang:
Whenever an OTP process (e.g. gen_server, gen_event) crashes, the reason
for the crash is meticulously logged by the OTP logging framework. This is
pretty much the only way to debug asynchronous behaviors, as if the
exceptions are swallowed by default you have no outward indication of what
happened.
Erlang provides a great example of a system that is the exact opposite of
the one you are proposing.
On Thu, Jul 5, 2012 at 10:26 AM, rex goxman <lists@ruby-forum.com> wrote:
1. Tell that to Erlangers.
--
Tony Arcieri
Xavier Noria писал 04.07.2012 16:41:
Ruby 1.9 uses kernel threads and does not have green threads.
There is nothing like Erlang's lightweight processes in Ruby (or, for
that matter, in most of the languages).That is clear, but the definition of green thread depends on who schedules
them.Is the GIL acting as a scheduler? Has MRI logic about what to lock next
therefore scheduling in practice *among the threads in its process*? Or is
my conjecture right that locking is mostly dumb and that ultimately the
kernel is the scheduler because the thread is native?The OP's problem is not the scheduler itself but the associated system
resources.
You can create 20k Erlang processes ("threads"). You cannot create 20k
pthreads. Just that.
Looks so also to me, but Ilya's post makes that clear (and it is a known
characteristic of MRI 1.9)... why the followup then after I linked to the
post?
On Wed, Jul 4, 2012 at 2:45 PM, Peter Zotov <whitequark@whitequark.org>wrote:
On Wed, Jul 4, 2012 at 2:30 PM, Peter Zotov <whitequark@whitequark.org>** >> wrote:
Jan E. wrote in post #1067435:
rex goxman wrote in post #1067433:
Well, for the purposes here, if Ruby's threads were green threads, yes
they would have been comparable to Erlang green processes. My emphasis
is far more on the "green" and far less on whether it is a process or a
thread.Ruby 1.8 *has* green threads. It's explained in Xavier's link, and I
also mentioned it above.You seem to misread "some implementations have native threads" as "all
have native threads".
The title of this post is "green threads in 1.9.* ?" Furthermore, I
have this edit in one of my posts here:
EDIT: 1.9.x of course.
So I have misread nothing. I am **ONLY** talking about 1.9.*. I could
care less about 1.8 or any other implementation.
Thanks.
--
Posted via http://www.ruby-forum.com/\.
MRI is still doing the scheduling in 1.9. It runs a timer thread that
interrupts running threads after a certain amount of time so waiting
threads can run. This way a single thread doing some computationally
intensive operation doesn't monopolize the entire VM.
On Wed, Jul 4, 2012 at 5:11 AM, Xavier Noria <fxn@hashref.com> wrote:
Then, behind the scenes, they are mapped to native threads in MRI 1.9 and
JRuby. One-to-one. In the case of MRI there's a GIL. I guess technically a
GIL does not mean that MRI is doing the scheduling, it means MRI is holding
a lock, but the kernel schedules the native thread (because it is native).
Maybe there's a gray area in the definition of green thread here, I don't
know.
--
Tony Arcieri
Tony Arcieri wrote in post #1067563:
Erlang provides a great example of a system that is the exact opposite
of
the one you are proposing.
1. I have proposed no system. I have stated that I'd like to use green
threads, and asked if ruby still had them available.
2. Erlang uses green processes, emphasis on 'green'.
--
Posted via http://www.ruby-forum.com/\.
Couldn't. Unless your level of caring is greater than your minimum. Or
you can have negative carativitiy.
I'm pretty sure Xavier (fxn) has answered this thread pretty well. My
question is: do you really care about the greenness of the threads
(academically), or do you actually want to know how best to utilise
them since they seem to have unusual profiling characteristics?
If the former, I don't know what to suggest. Research, code scouring,
reverse engineering (or read the link posted earlier). If the latter,
I say explore and research, and if you find something that works best
for your application use it, and maybe share it with us in case we
need to do it again some day.
As an aside, and I'm not pointing the finger at anyone in this thread
when I say this, but a week or two ago I commented to a colleague of
mine about how polite the ruby-talk forum was, and how people seemed
to consciously avoid flaming and trolling, and how noobs got useful
help even when they asked "dumb noob" questions, and how people said
"thank you" . . . but in the past couple of days there've been a few
discussions (usually related to optimisation or performance) that have
developed a very negative tone, if not outright ALL CAPS TYPING and
rage. What's going on? I want a return to the happy, friendly
ruby-talk of yesterweek.
On 5 July 2012 03:03, rex goxman <lists@ruby-forum.com> wrote:
So I have misread nothing. I am **ONLY** talking about 1.9.*. I could
care less about 1.8 or any other implementation.
--
Matthew Kerwin, B.Sc (CompSci) (Hons)
http://matthew.kerwin.net.au/
ABN: 59-013-727-651
"You'll never find a programming language that frees
you from the burden of clarifying your ideas." - xkcd
A more accurate picture of 1.9 is:
The OS schedules threads to run on CPUs.
The ruby VM has a global lock that only allows one thread to run ruby code at a time (the GVL). The GVL is also held for execution inside C extensions unless the author releases the GVL (most extensions release the GVL for some part of their operation).
Ruby schedules use of the GVL for threads created by the ruby VM.
A Ruby thread may release the GVL to run non-ruby code. An example would be performing a long calculation such as interacting with a DB, processing zlib streams, XML parsing with libxml2, etc. Such tasks don't need to communicate with the ruby VM so they can be running on separate CPUs at the same time as the ruby VM.
A Ruby thread will release the GVL when it performs a blocking operation such as IO or other system calls.
So the OS and the ruby VM cooperate on thread scheduling. If multiple ruby threads do not need the GVL to run the OS does the majority of the scheduling. If multiple ruby threads do need the GVL ruby does the majority of the scheduling.
On Jul 4, 2012, at 18:33, Tony Arcieri wrote:
On Wed, Jul 4, 2012 at 5:11 AM, Xavier Noria <fxn@hashref.com> wrote:
Then, behind the scenes, they are mapped to native threads in MRI 1.9 and JRuby. One-to-one. In the case of MRI there's a GIL. I guess technically a GIL does not mean that MRI is doing the scheduling, it means MRI is holding a lock, but the kernel schedules the native thread (because it is native). Maybe there's a gray area in the definition of green thread here, I don't know.
MRI is still doing the scheduling in 1.9. It runs a timer thread that interrupts running threads after a certain amount of time so waiting threads can run. This way a single thread doing some computationally intensive operation doesn't monopolize the entire VM.
1. I have proposed no system. I have stated that I'd like to use green
threads, and asked if ruby still had them available.
Your suggested use case for green threads was firing off background jobs
that may or may not break, with zero interest as to if they complete
successfully. This is not what Erlang does. Erlang/OTP provides extensive
reporting of all failures of any OTP components.
At least, that seems to be what you're suggesting here. Please correct me
if I'm wrong:
This is a crappy example, but say you need to do some
On Thu, Jul 5, 2012 at 12:31 PM, rex goxman <lists@ruby-forum.com> wrote:
calculation which might fail or even crash, but you don't want (or need)
to check the inputs to the calculation, or wrap the thing up in an error
catching mechanism, or anything of this nature. You just want to
calculate it, and if it works, fine, and if it doesn't (or even crashes)
- fine, your main program continues to run.
--
Tony Arcieri
This would make a good FAQ entry, I think. Thanks, Eric.
Jos
On Thu, Jul 5, 2012 at 1:22 PM, Eric Hodel <drbrain@segment7.net> wrote:
A more accurate picture of 1.9 is:
The OS schedules threads to run on CPUs.
The ruby VM has a global lock that only allows one thread to run ruby code
at a time (the GVL). The GVL is also held for execution inside C
extensions unless the author releases the GVL (most extensions release the
GVL for some part of their operation).Ruby schedules use of the GVL for threads created by the ruby VM.
A Ruby thread may release the GVL to run non-ruby code. An example would
be performing a long calculation such as interacting with a DB, processing
zlib streams, XML parsing with libxml2, etc. Such tasks don't need to
communicate with the ruby VM so they can be running on separate CPUs at the
same time as the ruby VM.A Ruby thread will release the GVL when it performs a blocking operation
such as IO or other system calls.So the OS and the ruby VM cooperate on thread scheduling. If multiple
ruby threads do not need the GVL to run the OS does the majority of the
scheduling. If multiple ruby threads do need the GVL ruby does the
majority of the scheduling.
--
Jos Backus
jos at catnook.com
Eric Hodel wrote in post #1067586:
Ruby schedules use of the GVL for threads created by the ruby VM.
A Ruby thread may release the GVL to run non-ruby code.
What is a ruby thread? A green thread? A kernel thread? A green
thread mapped to a kernel thread?
The depiction at the link somewhere above seems to suggest it is a green
thread which is mapped one-to-one to a kernel thread. Some here suggest
there is no such thing, just kernel threads. I will assume it is a
kernel thread unless you say otherwise.
Eric Wong wrote...
I know you don't like fibers, but the "neverblock" gem uses Fibers
transparently under 1.9 to provide green threads. However, I'm
not sure how active "neverblock" development is these days.
Actually I like fibers quite a bit, I just didn't think they were
applicable in this particular circumstance. I can check into neverblock
- thanks.
--
Posted via http://www.ruby-forum.com/\.
I'd also like to reiterate that Erlang is doing the right thing here and
that silently swallowing errors in an asynchronous concurrent system (which
is what you're proposing) is an extremely bad idea. In an asynchronous
system, the error log is your only tool for understanding what's broken and
where. You will most certainly want to capture and log all errors, not
silently ignore them, or else you will have no visibility into if your
asynchronous background tasks are working or not.
On Thu, Jul 5, 2012 at 5:37 PM, Tony Arcieri <tony.arcieri@gmail.com> wrote:
On Thu, Jul 5, 2012 at 12:31 PM, rex goxman <lists@ruby-forum.com> wrote:
1. I have proposed no system. I have stated that I'd like to use green
threads, and asked if ruby still had them available.Your suggested use case for green threads was firing off background jobs
that may or may not break, with zero interest as to if they complete
successfully. This is not what Erlang does. Erlang/OTP provides extensive
reporting of all failures of any OTP components.At least, that seems to be what you're suggesting here. Please correct me
if I'm wrong:This is a crappy example, but say you need to do some
calculation which might fail or even crash, but you don't want (or need)
to check the inputs to the calculation, or wrap the thing up in an error
catching mechanism, or anything of this nature. You just want to
calculate it, and if it works, fine, and if it doesn't (or even crashes)
- fine, your main program continues to run.--
Tony Arcieri
--
Tony Arcieri
Tony Arcieri wrote in post #1067609:
Your suggested use case for green threads was firing off background jobs
that may or may not break, with zero interest as to if they complete
successfully.
I'm not sure if you are someone with reading-comprehension issues, or a
troll. I will state (again) that I threw a "crappy" example out there
off the top of my head as a reason someone might want green threads. I
did this because some people seem to lack the creativity to come up with
use cases themselves, which are plentiful and abound. In hindsight, it
seems to have been a serious mistake on my part to have done this, since
you continue to ignore the fact that I stated it was a "crappy" example
and continue to inexplicably seize upon what I wrote as some opportunity
to "hammer" me.
Having said that, I stand by what I said. You might want to fire off
background jobs that may or may not break, with zero interest as to if
they complete successfully.
This is not what Erlang does. Erlang/OTP provides
extensive
reporting of all failures of any OTP components.
Erlang "does" whatever you want it to do, bro. If you want process
supervision, extensive reporting of all failures, blah blah, it is
there. If you don't want that (and in many cases, you don't), you don't
have to use it. As with anything, it depends on your application and
what you want to do with it.
--
Posted via http://www.ruby-forum.com/\.
I want a return to the happy, friendly ruby-talk of yesterweek.
One small step would be to stop (gently) mocking people for using
well-accepted variants of colloquial idioms. 
I could care less about 1.8 or any other implementation.
Couldn't. Unless your level of caring is greater than your minimum. Or you can have negative carativitiy.
http://articles.boston.com/2010-10-24/lifestyle/29303907_1_care-peeves-decades
http://itre.cis.upenn.edu/~myl/languagelog/archives/001182.html
TL;DR: "I could care less" is at least 50 years old, and can be
explained as sarcasm, hyperbole, and/or a long-ago confusion now lost
in history (like "una narancia" became "a norange" became "an orange")
-- not as evidence of the speaker's idiocy.
That said, rex was being pretty defensive and peevish. Group hug!
--
Alex Chaffee - alex@stinky.com
http://alexchaffee.com
http://twitter.com/alexch
Tony Arcieri wrote in post #1067610:
I'd also like to reiterate that Erlang is doing the right thing here and
that silently swallowing errors in an asynchronous concurrent system
(which
is what you're proposing) is an extremely bad idea.
If you are saying that there never has been nor never will be a use case
for anyone on the planet where they might want to silently swallow
errors in an asynchronous concurrent system, and that to even
contemplate such is always a bad idea, then you, sir, are an idiot.
--
Posted via http://www.ruby-forum.com/\.
So in other words, you don't have a use case for green threads
On Thu, Jul 5, 2012 at 8:24 PM, rex goxman <lists@ruby-forum.com> wrote:
I'm not sure if you are someone with reading-comprehension issues, or a
troll. I will state (again) that I threw a "crappy" example out there
off the top of my head as a reason someone might want green threads. I
did this because some people seem to lack the creativity to come up with
use cases themselves, which are plentiful and abound. In hindsight, it
seems to have been a serious mistake on my part to have done this, since
you continue to ignore the fact that I stated it was a "crappy" example
and continue to inexplicably seize upon what I wrote as some opportunity
to "hammer" me.
--
Tony Arcieri
Eric Hodel wrote in post #1067586:
Ruby schedules use of the GVL for threads created by the ruby VM.
A Ruby thread may release the GVL to run non-ruby code.
For these answers I am ignoring Windows. You can probably treat "POSIX thread" as the equivalent userland threads on Windows, but don't take my word for it.
What is a ruby thread?
A thread spawned from ruby. Non-ruby threads can be spawned by other libraries. For example, if you've embedded ruby in a game engine the game engine can have many of its own threads which ask ruby threads to do work.
A green thread?
Any program that manages thread creation and scheduling for itself without use of the OS uses green threads.
A kernel thread?
The most literal definition is a thread that runs in, and only in, the kernel. You can ignore them for most purposes since you don't have any direct control over when or how many are created by your process (this is true for all programs, not just ruby programs). I think that most frequently, kernel threads are mapped 1:1 to userland threads, but this may vary between different OSs.
At one point FreeBSD had multiple pthread implementations you could link in your program. One had 1:1 mapping between kernel threads and userland threads, one had 1:N mapping (one kernel thread per process) and one M:N mapping (multiple userland threads per kernel thread).
Userland threads are what userland programs run. The POSIX thread API (pthread) handles userland threads and takes care of spawning the threads, allocating stack space for them, etc. Depending upon the mapping to kernel threads they may be scheduled from userland or from the kernel. (I think, most frequently, from the kernel.)
A green thread mapped to a kernel thread?
Since a green thread is scheduled from within a program the kernel has no knowledge of it so they won't be mapped.
The depiction at the link somewhere above seems to suggest it is a green
thread which is mapped one-to-one to a kernel thread.
Since pthreads typically make some communication with the kernel it isn't correct to call them green threads. It's possible to have a pthread library that makes no communication with the kernel (a library similar to FreeBSD's 1:N thread library could do this), but in practice I doubt this happens. It's difficult for a thread library to schedule a process' threads on multiple CPUs without some communication with the kernel.
Some here suggest there is no such thing, just kernel threads. I will assume it is a kernel thread unless you say otherwise.
I think what they describe as "kernel threads" I call "userland threads". The OS often has its own threads to perform OS tasks such as userland scheduling, handling interrupts, polling for IO from hardware, etc. I don't wish to confuse the two.
Digging in further to thread.c, the GVL is not directly part of userland thread scheduling or ruby thread scheduling, it's just a lock around the ruby VM that allows a thread to use the ruby VM. (If a thread calls methods in the Ruby VM without the GVL it will likely cause corruption and crashes.) Ruby uses pthread API to switch threads so it's up to the OS which threads get run.
Ruby will interrupt threads that have run for "too long" and allow the OS to schedule another thread. This is all performed through pthread mutexes and condition variables. (Perhaps I'm missing it, but I'm no longer seeing a list of threads waiting to run as I recall 1.8 having which is a requirement for a green thread implementation.)
On Jul 5, 2012, at 14:22, rex goxman wrote:
Do you HAVE to have the last word? Is that what this is?
There are plenty of use cases for green threads, and YOU know that. The fact that Rex is not telling you his means nothing. Stop being a jackass.
On Jul 5, 2012, at 20:55 , Tony Arcieri wrote:
On Thu, Jul 5, 2012 at 8:24 PM, rex goxman <lists@ruby-forum.com> wrote:
I'm not sure if you are someone with reading-comprehension issues, or a
troll. I will state (again) that I threw a "crappy" example out there
off the top of my head as a reason someone might want green threads. I
did this because some people seem to lack the creativity to come up with
use cases themselves, which are plentiful and abound. In hindsight, it
seems to have been a serious mistake on my part to have done this, since
you continue to ignore the fact that I stated it was a "crappy" example
and continue to inexplicably seize upon what I wrote as some opportunity
to "hammer" me.So in other words, you don't have a use case for green threads
I'm not sure if you are someone with reading-comprehension issues, or a
troll. I will state (again) that I threw a "crappy" example out there
off the top of my head as a reason someone might want green threads. I
did this because some people seem to lack the creativity to come up with
use cases themselves, which are plentiful and abound. In hindsight, it
seems to have been a serious mistake on my part to have done this, since
you continue to ignore the fact that I stated it was a "crappy" example
and continue to inexplicably seize upon what I wrote as some opportunity
to "hammer" me.So in other words, you don't have a use case for green threads
Do you HAVE to have the last word? Is that what this is?
I think the point is, that as long as we do not know rex's real use
case we cannot come up with proper suggestions for alternative
solutions to green threads. So far rex provides a solution he wants
to use but I cannot really see the reasoning behind it.
Btw. even green threads come at a price and firing off a lot of tasks
by just creating threads (whatever color) when the task appears is not
likely going to work out well for large numbers of concurrent tasks.
In this case one would need some form of thread pool with a queue
anyway. And then the difference between green and kernel thread
creation is irrelevant anyway.
There are plenty of use cases for green threads, and YOU know that. The fact that Rex is not telling you his means nothing. Stop being a jackass.
Actually I have doubts that there are so many use cases for green
threads. Off the top of my head only platforms which do not support
kernel threads come to mind. I'd love to hear more and especially the
case rex has in mind.
Other than that the overhead in terms of memory and CPU for creating a
kernel thread isn't too big nowadays. And from a global perspective
which covers operating system as well as hardware trends Matz's
decision to go for kernel threads is wise because that will eventually
give us better performance (once GIL is gone).
Kind regards
robert
On Fri, Jul 6, 2012 at 8:52 AM, Ryan Davis <ryand-ruby@zenspider.com> wrote:
On Jul 5, 2012, at 20:55 , Tony Arcieri wrote:
On Thu, Jul 5, 2012 at 8:24 PM, rex goxman <lists@ruby-forum.com> wrote:
--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/