I think the biggest issue is not threads per se, but rather threads
with shared state. A non-shared-state thread could be preferable
to an OS process.
-mental
On Thu, 4 Oct 2007 06:05:09 +0900, Phlip <phlip2005@gmail.com> wrote:
Daniel DeLorme wrote:
+1
threads are a misfeatureThreads are worse than GOTO.
Speaking from my AOL experiences (which are admittedly getting a bit long
in the tooth compared to today's apps and hardware):
Yeah, but.
On the one hand, our hardware costs certainly dwarfed our development
costs. (And by "certainly", I mean "I certainly imagine so, having
absolutely no memory of what the actual numbers were.") And our operating
costs were a big part of those.
What's more, cooling and power do NOT scale linearly; at some point, the
incremental cost of one more CPU is a new data center or an extra utility
trench. Plus, at the time, anyway, there weren't big enough power/cooling
products on the market to handle our needs. I imagine that's changed now.
On the other hand, that wasn't our biggest problem. If handling ten
million users costs $X/user, and makes $Y/user profit, and doubling to 20
million users still costs roughly $X/user and thus makes $Y/user profit,
then doubling capacity to double the user base is really easy.
Hardware/operating costs do impact whether you can make a per-user profit
in the first place, but they don't impact scaling costs. Whether Ruby
takes up too much server power is a base hardware cost, not a scaling cost.
If it's too heavyweight, it's too heavyweight at 1000 users, just like it
will be at 100 million.
But our biggest problem was development - not even development costs, but
development *ability*. Software development does not scale anywhere NEAR
linearly, as Frederick Brooks well knows. Double the number of
programmers, and you've exponentially increased the number of interpersonal
communications and meetings. That needs more project managers and
technical managers, which again increases the number of communications
channels. And that needs more support staff (HR, legal, facilities, etc.)
More people need more office space and parking, and the bigger your campus,
the more time is wasted travelling, and the more space is "wasted" on
non-productive square footage like kitchens and hallways. And so forth.
I haven't even mentioned the sheer difficulty of hiring that many
developers. I imagine many here have had the problem of "If I could hire
someone else, I'd have less to do; but I have too much to do to find time
to hire someone." That doesn't scale either. Then there's saturation. At
one point, there were nearly as many technical job openings in the
Washington Post as there were total unemployed people in the Washington
area. Bad news.
Then there's the sad fact that software simply does not scale cleanly.
Even if the core functions are dollar-scalable - and I'll touch on that in
another post - things like maintenance, operability, reporting, metering,
debuggability, etc. are not. So as your business scales, you'll spend more
and more time doing infrastructure development instead of feature
development. There comes a point where there's a negative economy of
scale; you're too big to actually do any real innovation, because all your
resources are sucked up keeping the beast alive.
The point of all this is that anything - ANYTHING - that increases
developer productivity is a huge, huge win. Sure, developer salaries may
not have been over 50% of our expenditures. But without developer
productivity, you stagnate, and someone else zooms right past you.
On Wed, 3 Oct 2007 22:19:57 +0900, Charles Oliver Nutter wrote:
I find this perspective puzzling. In most large datacenters, the big
cost of operation is neither the cost of the servers nor the cost of the
development time to put code on them, it's the peripheral electricity,
administration and cooling costs once the application written must be
deployed to thousands of users.
--
Jay Levitt |
Boston, MA | My character doesn't like it when they
Faster: jay at jay dot fm | cry or shout or hit.
http://www.jay.fm | - Kristoffer
Brad Phelan wrote:
With regards to Python/Ruby speed issues. I am currently watching a
complex scheduling algorithm ticking by very slowly in a shell while I
write this post. The algorithm should should have been written in C/C++
and not in Python/Psyco or Ruby and I am going nuts waiting for it to
complete.( BTW I didn't write it )
"Complex scheduling algorithm" means different things to different
people. Is it slow because the algorithm sucks or slow because it's not
written in C/C++? What kind of scheduling is it -- combinatorial?
He decided to write it _himself_. That's the main piece.
On Sep 27, 2007, at 8:25 AM, Chad Perrin wrote:
How do you know that was the reason? The impression I got from the
lengthy, slashdotted explanation was that the project was unfinished
after two years, so he decided to junk it and start over in PHP.
Thank you!! It's about time somebody put a dollar figure on the cost of
poor scalability and highlighted the nonsense of "adding servers is
cheaper than hiring programmers." They are two entirely different
economic propositions.
That's true. However, very roughly, compute resource can scale about
linearly with compute requirement.
Alternatively, you can reduce the compute requirement by having a more
complex software system. However, the amount of programmer needed to
build and maintain a given complexity of software certainly doesn't
scale linearly with the system complexity (see the mythical man month).
I'm also not a fan of throwing hardware at a problem as a cure all, and
I don't like dumb "enterprise" grade solutions when there are much more
powerful alternatives. But I'm also not a fan of automatically assuming
that you need to build lots of clever and complex things, and that
existing components just wont do the job. Pragmatism seems like the best
approach 
> I find this perspective puzzling. In most large datacenters, the big
> cost of operation is neither the cost of the servers nor the cost of the
> development time to put code on them, it's the peripheral electricity,
> administration and cooling costs once the application written must be
> deployed to thousands of users."Most"? If you define "large data center" as the very top echelon,
then maybe, but even then I'd like to see some data. I expect the vast
majority (all?) of readers of this ng will be involved in scenarios in
which the cost of development time far exceeds electricity or server
costs for their deployed applications.Part of what kept me from getting involved in Ruby sooner than I did
was my erroneous view that I wanted to be using technology that would
be sufficient to run Amazon, Ebay, etc. Little did it matter that I
wasn't pursuing that type of project - analogous to the fact that
most, if not all, Hummer drivers will never encounter serious off road
or combat situations
That's okay, though, because most civilian Hummers are unsuited to
offroading and combat situations.
I'm all for increasing the performance and scalability of Ruby, but I
think the productivity gains still outweigh the extra runtime costs
for most projects.
Agreed -- for the vast majority of projects in the range for which Ruby
is typically used, as compared with languages with execution performance
that is actually sufficiently better than Ruby's to bother measuring the
difference. I mean, sure, it's fun to compare benchmarks between Python
and Ruby (for instance), but if you actually need a performance boost you
should be comparing Ruby with C instead (or comparing Python with C, as
your tastes in productivity-enhancing languages may lead you).
On Thu, Oct 04, 2007 at 02:15:09AM +0900, Brian Adkins wrote:
On Oct 3, 9:19 am, Charles Oliver Nutter <charles.nut...@sun.com> > wrote:
--
CCD CopyWrite Chad Perrin [ http://ccd.apotheon.org ]
They always say that when life gives you lemons you should make lemonade.
I always wonder -- isn't the lemonade going to suck if life doesn't give
you any sugar?
M. Edward (Ed) Borasky wrote:
Brad Phelan wrote:
With regards to Python/Ruby speed issues. I am currently watching a
complex scheduling algorithm ticking by very slowly in a shell while I
write this post. The algorithm should should have been written in C/C++
and not in Python/Psyco or Ruby and I am going nuts waiting for it to
complete.( BTW I didn't write it )
"Complex scheduling algorithm" means different things to different
people. Is it slow because the algorithm sucks or slow because it's not
written in C/C++? What kind of scheduling is it -- combinatorial?
I have never looked at the algorithm myself. It is a TDMA message scheduling algorithm. Given a set of messages on a time partitioned bus, with multiple transmitters and receivers find the optimal message schedule. It is not too different from the classic M$ Project scheduling really. However you have thousands of messages to schedule and many constraints. Still I am not sure that waiting 2 minutes to several hours for a schedule to complete would occur if exactly the same algorithm was written in C rather than Python. This is pure speculation however as I've never had time to look over it myself and make a proper judgment.
However the argument from the developers is that real customers of the tool rarely run a schedule. Once the schedule has been fixed it is rarely changed. Therefore the pain of waiting for 5 minutes or even a few hours in the case of a large schedule is little pain in the overall scheme of a project that may last many years.
The only reason it bugs me is that I am constantly running schedules to generate test cases for some other downstream code I am running. It therefore causes me pain in ways it would not necessarily do for a real customer. This brings us round to the original argument I guess. Perhaps in this case Python is suitable. It is easy to code, and easy to analyze, debug and maintain. The slowness is not a huge factor because the customer just doesn't require it to be "fast"
B
Yes, he decided to write it himself -- after giving up on Rails, for
reasons that, as far as I'm aware, relate to the fact that it wasn't done
in Rails after two years.
On Thu, Sep 27, 2007 at 11:10:17PM +0900, julik wrote:
On Sep 27, 2007, at 8:25 AM, Chad Perrin wrote:
>How do you know that was the reason? The impression I got from the
>lengthy, slashdotted explanation was that the project was unfinished
>after two years, so he decided to junk it and start over in PHP.He decided to write it _himself_. That's the main piece.
--
CCD CopyWrite Chad Perrin [ http://ccd.apotheon.org ]
Baltasar Gracian: "A wise man gets more from his enemies than a fool from
his friends."
Thank you!! It's about time somebody put a dollar figure on the cost of
poor scalability and highlighted the nonsense of "adding servers is
cheaper than hiring programmers." They are two entirely different
economic propositions.That's true. However, very roughly, compute resource can scale about
linearly with compute requirement.
What about Amdahl's law?
Alternatively, you can reduce the compute requirement by having a more
complex software system.
While it's true that very simple systems can perform badly because
they use poor algorithms and/or do not make dynamic optimizations,
more complex software generally means increased computational
requirements.
-mental
On Thu, 4 Oct 2007 00:19:40 +0900, benjohn@fysh.org wrote:
Chad Perrin wrote:
Yes, he decided to write it himself -- after giving up on Rails, for
reasons that, as far as I'm aware, relate to the fact that it wasn't
done in Rails after two years.
Actually, if I can be allowed to read between the lines, he went back
when his Ruby mentor left and he realized that he was not able to do it
in Ruby. He went back to what he knew when he was left on his own. He
sort of says that in the post.
--
Posted via http://www.ruby-forum.com/\.
>> Thank you!! It's about time somebody put a dollar figure on the cost of
>> poor scalability and highlighted the nonsense of "adding servers is
>> cheaper than hiring programmers." They are two entirely different
>> economic propositions.
>
> That's true. However, very roughly, compute resource can scale about
> linearly with compute requirement.What about Amdahl's law?
What about it? Unless you're writing software that doesn't scale with
the hardware, more hardware means linear scaling, assuming bandwidth
upgrades. If bandwidth upgrades top out, you've got a bottleneck no
amount of hardware purchasing or programmer time will ever solve.
> Alternatively, you can reduce the compute requirement by having a more
> complex software system.While it's true that very simple systems can perform badly because
they use poor algorithms and/or do not make dynamic optimizations,
more complex software generally means increased computational
requirements.
I thought "complex" was a poor choice of term here, for the most part.
It was probably meant as a stand-in for "more work at streamlining
design, combined with greater code cleverness needs to scale without
throwing hardware at the problem."
On Thu, Oct 04, 2007 at 12:38:25AM +0900, MenTaLguY wrote:
On Thu, 4 Oct 2007 00:19:40 +0900, benjohn@fysh.org wrote:
--
CCD CopyWrite Chad Perrin [ http://ccd.apotheon.org ]
Leon Festinger: "A man with a conviction is a hard man to change. Tell him
you disagree and he turns away. Show him facts and figures and he questions
your sources. Appeal to logic and he fails to see your point."
Well . . . yes, but judging by the phrasing he considers the main reason
for switching back to PHP to be that Rails didn't get the job done in two
years of development. He may well have stuck with Rails longer if the
"Ruby mentor" hadn't departed for greener pastures, but a lot of effort
seems to be spent in that piece on pointing out that he got done in a
very short time what Rails hadn't allowed in two years.
Note that I'm speaking of what the "analysis" seems to be saying, and not
my own opinions of what Rails can or cannot do. It seems ridiculous to
me that someone couldn't build a working web app in two years, regardless
of the tool used in the development effort (as I've already said).
On Fri, Sep 28, 2007 at 01:43:03AM +0900, Lloyd Linklater wrote:
Chad Perrin wrote:
> Yes, he decided to write it himself -- after giving up on Rails, for
> reasons that, as far as I'm aware, relate to the fact that it wasn't
> done in Rails after two years.Actually, if I can be allowed to read between the lines, he went back
when his Ruby mentor left and he realized that he was not able to do it
in Ruby. He went back to what he knew when he was left on his own. He
sort of says that in the post.
--
CCD CopyWrite Chad Perrin [ http://ccd.apotheon.org ]
Baltasar Gracian: "A wise man gets more from his enemies than a fool from
his friends."
> That's true. However, very roughly, compute resource can scale about
> linearly with compute requirement.What about Amdahl's law?
What about it? Unless you're writing software that doesn't scale with
the hardware, more hardware means linear scaling, assuming bandwidth
upgrades. If bandwidth upgrades top out, you've got a bottleneck no
amount of hardware purchasing or programmer time will ever solve.
Amdahl's law is relevant because most software _can't_ be written to
scale entirely linearly with the hardware, because most computational
problems are limited in the amount of parallelism they admit. You may
have been fortunate enough to have been presented with a lot of
embarrassingly parallel problems to solve, but that isn't the norm.
> Alternatively, you can reduce the compute requirement by having a more
> complex software system.While it's true that very simple systems can perform badly because
they use poor algorithms and/or do not make dynamic optimizations,
more complex software generally means increased computational
requirements.I thought "complex" was a poor choice of term here, for the most part.
It was probably meant as a stand-in for "more work at streamlining
design, combined with greater code cleverness needs to scale without
throwing hardware at the problem."
No argument there, as long as it's understood that there are limits to
what can be achieved. I don't want to discourage anyone from seeking
linear scalability as an ideal, but it's not a realistic thing to
promise or assume.
-mental
On Thu, 4 Oct 2007 01:42:22 +0900, Chad Perrin <perrin@apotheon.com> wrote:
Actually, this is not entirely true. Amdahl's law only applies to
optimizing a fixed workload. I need to rethink this argument.
-mental
On Thu, 4 Oct 2007 02:39:28 +0900, MenTaLguY <mental@rydia.net> wrote:
On Thu, 4 Oct 2007 01:42:22 +0900, Chad Perrin <perrin@apotheon.com> wrote:
> That's true. However, very roughly, compute resource can scale about
> linearly with compute requirement.What about Amdahl's law?
What about it? Unless you're writing software that doesn't scale with
the hardware, more hardware means linear scaling, assuming bandwidth
upgrades. If bandwidth upgrades top out, you've got a bottleneck no
amount of hardware purchasing or programmer time will ever solve.Amdahl's law is relevant because most software _can't_ be written to
scale entirely linearly with the hardware, because most computational
problems are limited in the amount of parallelism they admit. You may
have been fortunate enough to have been presented with a lot of
embarrassingly parallel problems to solve, but that isn't the norm.
>> > That's true. However, very roughly, compute resource can scale about
>> > linearly with compute requirement.
>>
>> What about Amdahl's law?
>
> What about it? Unless you're writing software that doesn't scale with
> the hardware, more hardware means linear scaling, assuming bandwidth
> upgrades. If bandwidth upgrades top out, you've got a bottleneck no
> amount of hardware purchasing or programmer time will ever solve.Amdahl's law is relevant because most software _can't_ be written to
scale entirely linearly with the hardware, because most computational
problems are limited in the amount of parallelism they admit. You may
have been fortunate enough to have been presented with a lot of
embarrassingly parallel problems to solve, but that isn't the norm.
Maybe not "entirely", but certainly close enough for government (or
corporate) work. I was under the impression we were talking about
massive-traffic server-based systems here, where throwing more hardware
at the problem (in the sense of extra blades, or whatever) is an option.
I did not think we were talking about something like a desktop app where
opportunities for parallelism are strictly limited -- in which case I'd
agree that throwing more hardware at the problem is a non-starter. Of
course, I don't know anyone who thinks endlessly adding processors to
your desktop system is the correct answer to a slow word processor.
>> > Alternatively, you can reduce the compute requirement by having a more
>> > complex software system.
>>
>> While it's true that very simple systems can perform badly because
>> they use poor algorithms and/or do not make dynamic optimizations,
>> more complex software generally means increased computational
>> requirements.
>
> I thought "complex" was a poor choice of term here, for the most part.
> It was probably meant as a stand-in for "more work at streamlining
> design, combined with greater code cleverness needs to scale without
> throwing hardware at the problem."No argument there, as long as it's understood that there are limits to
what can be achieved. I don't want to discourage anyone from seeking
linear scalability as an ideal, but it's not a realistic thing to
promise or assume.
It's close enough (again), for many purposes, to "realistic". When you
can get roughly linear scaling up to 100 times as much scaling needs, as
opposed to trying to get similar scaling capabilities out of throwing
programmers (or programmer time) at the problem, that's certainly
"realistic" in my estimation.
Obviously I'm not saying that you should write crap code and throw
hardware at it. On the other hand, there's a sweet spot for effort spent
in developing good, performant code -- and beyond that point, you should
consider throwing hardware at the problem. In such circumstances, one of
the primary measures of quality code is "Does it scale in a roughly
linear manner when you add compute resources?"
On Thu, Oct 04, 2007 at 02:39:28AM +0900, MenTaLguY wrote:
On Thu, 4 Oct 2007 01:42:22 +0900, Chad Perrin <perrin@apotheon.com> wrote:
--
CCD CopyWrite Chad Perrin [ http://ccd.apotheon.org ]
MacUser, Nov. 1990: "There comes a time in the history of any project when
it becomes necessary to shoot the engineers and begin production."
Maybe not "entirely", but certainly close enough for government (or
corporate) work. I was under the impression we were talking about
massive-traffic server-based systems here, where throwing more hardware
at the problem (in the sense of extra blades, or whatever) is an option.
I did not think we were talking about something like a desktop app where
opportunities for parallelism are strictly limited -- in which case I'd
agree that throwing more hardware at the problem is a non-starter. Of
course, I don't know anyone who thinks endlessly adding processors to
your desktop system is the correct answer to a slow word processor.
I wasn't thinking of only desktop systems, but I think you're right:
many massive-traffic server-based systems _can_ be embarrassingly
parallel, since jobs are often relatively independent (e.g. individual
user sessions/HTTP requests), and in that context adding more work
usually translates into simply adding more jobs. It's going to depend
a lot on the nature of the problem domain, though: once the jobs
become sufficiently interdependent, you do start to hit a scalability
wall (often a problem in e.g. simulations or online games).
Anyway, I think I was wrong: Amdahl's law may not be appropriate
here -- we aren't just talking about making a fixed-size job go
faster, but what happens when more work is added. Unless you're doing
something like n-body simulations, the relative amount of
unparallelizable work can decrease as the absolute amount of work
increases, because the shared stuff that forces serialization can often
be partitioned, allowing more overall parallelism than was practical
with a smaller workload.
It's close enough (again), for many purposes, to "realistic". When you
can get roughly linear scaling up to 100 times as much scaling needs, as
opposed to trying to get similar scaling capabilities out of throwing
programmers (or programmer time) at the problem, that's certainly
"realistic" in my estimation.
I'd submit that while you're still able to get a two-orders-magnitude
increase in performance from simply improving your algorithms, you're
probably not to the point where you could scale linearly. On the other
hand, by the time you're scaling linearly, there's probably not much
else to squeeze out of the thing, aside from micro-optimizations which
may get you some fractions of an order of magnitude improvement.
Obviously I'm not saying that you should write crap code and throw
hardware at it. On the other hand, there's a sweet spot for effort spent
in developing good, performant code -- and beyond that point, you should
consider throwing hardware at the problem. In such circumstances, one of
the primary measures of quality code is "Does it scale in a roughly
linear manner when you add compute resources?"
Yes, agreed.
-mental
On Thu, 4 Oct 2007 04:40:52 +0900, Chad Perrin <perrin@apotheon.com> wrote:
A lot depends on your application requirements. If you design it from the
ground up to be "shared nothing", then you may well be lucky enough to
truly HAVE shared nothing. But you'll also have a pretty limited feature
set.
What's the big buzzword today? Social networking. What did we used to
call that? "Community." What was the single biggest sticky-paper
community feature? Buddy lists. Who does buddy lists besides the Big Guys
(who can throw money at it) and the really small guys (who fit on a single
server)? Nobody. Why? Doesn't scale linearly. Think about what it takes
to offer a feature that, for every simultaneous user, checks the list of
every other simultaneous user for people you know. Shared-nothing *that*.
My area of expertise was the AOL mail system. And, looking back, there
were a number of core features we offered that simply couldn't be done in a
shared-nothing world over slow phone lines:
- You could see when each recipient had read your e-mail.
- If nobody had read it, you could unsend it.
- You could forward large attachments and long threads without re-uploading
them.
- Corollary: the server could handle large attachments and long threads
without storing multiple copies. (Disks were tiny then.)
- When sending e-mail, the system would check if all your recipients were
valid, not full, accepting e-mail from you, etc. If any were not, the
message wouldn't send. No bounces. (This requires a two-phase commit.)
- If the sender or other recipeients of an e-mail were online, their
address would become a hyperlink so you could IM them. (Buddy lists
again.)
- Your outbox pointed to the same message body as your recipients' inbox.
And all recipients' inboxes pointed to the same message as well. (Disk
space again.)
- The same e-mail message would appear differently to different clients,
depending on the feature set of that client.
- If you were the BCC recipient of an e-mail, you'd see a BCC header with
your name. If you were the author, you'd see all the BCCs.
- Large bulk mailings stored only a single copy of the message.
- E-mail complaints could be sent to us in a manner that preserved their
evidentiary value in court.
- The client stayed in a wait state after sending until the servers could
guarantee that your e-mail had actually been delivered. (Two-phase commit
again.)
That's just off the top of my head; I'm sure there were dozens of other
things I've forgotten that were designed when all the mail servers fit on
one machine, and then had to scale to multiple replicated data centers.
Could e-mail live without these features? Sure. Internet e-mail never had
them, a generation grew up without them, and these days nobody bemoans the
fact that you can't instantly know your message was delivered to its
destination; in fact, even bounces are becoming a thing of the past. If
you mistype an address, you may never know, and that's just the way it
works. "Did you get my e-mail?" is a real question, not just a
passive-aggressive way of saying "I see you read my message, but have not
yet responded."
And some of the features were only important in an age where pipes (both
last-mile and LAN) were very narrow and disks and RAM were very small.
Spam, in particular, made the "one copy of each message" model obsolete,
because spammers wouldn't play by the rules.
But restricting yourself to only shared-nothing features means ruling out
an awful lot of features. Including anything depending on a database
index, or a table that fits completely in memory, or any sort of
rate-limiting or duplicate-detection or spam prevention, or in fact
anything that makes any assumptions at all about the state of any database
you're interacting with or relational integrity or any other transaction in
the system, ever. Including whether the disk drive holding the transaction
you just wrote to disk has disappeared in a puff of head crash.
It was always the little things that bit us. Know why AOL screen names are
often "Jim293852"? Well, it started out as "The name 'Jim' is already
taken. Would you like 'Jim2'?". Guess how well that scales when the first
available Jim is "Jim35000"? Not very.
Pop-quiz: Which of *your* core features would you have to eliminate with
three million simultaneous users?
On Thu, 4 Oct 2007 04:40:52 +0900, Chad Perrin wrote:
No argument there, as long as it's understood that there are limits to
what can be achieved. I don't want to discourage anyone from seeking
linear scalability as an ideal, but it's not a realistic thing to
promise or assume.It's close enough (again), for many purposes, to "realistic". When you
can get roughly linear scaling up to 100 times as much scaling needs, as
opposed to trying to get similar scaling capabilities out of throwing
programmers (or programmer time) at the problem, that's certainly
"realistic" in my estimation.
--
Jay Levitt |
Boston, MA | My character doesn't like it when they
Faster: jay at jay dot fm | cry or shout or hit.
http://www.jay.fm | - Kristoffer
> It's close enough (again), for many purposes, to "realistic". When you
> can get roughly linear scaling up to 100 times as much scaling needs, as
> opposed to trying to get similar scaling capabilities out of throwing
> programmers (or programmer time) at the problem, that's certainly
> "realistic" in my estimation.I'd submit that while you're still able to get a two-orders-magnitude
increase in performance from simply improving your algorithms, you're
probably not to the point where you could scale linearly. On the other
hand, by the time you're scaling linearly, there's probably not much
else to squeeze out of the thing, aside from micro-optimizations which
may get you some fractions of an order of magnitude improvement.
Algorithms are important to performance, of course, and there's a minimal
amount of attention you should always want to employ in writing quality
code. My point is that ultimately, as things continue to scale upward,
you tend to pass a point where algorithm tweaking helps sufficiently any
longer to bother with it very much at about the same time you notice that
no matter what else you do you're going to have to add more hardware
resources.
> Obviously I'm not saying that you should write crap code and throw
> hardware at it. On the other hand, there's a sweet spot for effort spent
> in developing good, performant code -- and beyond that point, you should
> consider throwing hardware at the problem. In such circumstances, one of
> the primary measures of quality code is "Does it scale in a roughly
> linear manner when you add compute resources?"Yes, agreed.
It seems that on one hand I've been advocating writing code so you can
scale upward in hardware resources, and that throwing hardware at the
problem is eventually the only likely option you have left, while on the
other hand you're advocating for people writing good code in the first
place because crap code of sufficiently bad quality won't scale very well
no matter how much hardware you throw at it. In other words, I was
assuming reasonably good code as a baseline, and you were assuming
reasonably bad code as a baseline. These incompatible assumptions may be
influenced by our respective work environments.
I'll expand on my position, then:
1. Hire good programmers.
2. Have them write good code.
3. Throw hardware at the scaling problem, because your good code
written by good programmers can handle it.
On Thu, Oct 04, 2007 at 06:09:02AM +0900, MenTaLguY wrote:
On Thu, 4 Oct 2007 04:40:52 +0900, Chad Perrin <perrin@apotheon.com> wrote:
--
CCD CopyWrite Chad Perrin [ http://ccd.apotheon.org ]
Larry Wall: "A script is what you give the actors. A program is what you
give the audience."
Jay Levitt wrote:
Pop-quiz: Which of *your* core features would you have to eliminate with
three million simultaneous users?
R&D vacations
>> No argument there, as long as it's understood that there are limits to
>> what can be achieved. I don't want to discourage anyone from seeking
>> linear scalability as an ideal, but it's not a realistic thing to
>> promise or assume.
>
> It's close enough (again), for many purposes, to "realistic". When you
> can get roughly linear scaling up to 100 times as much scaling needs, as
> opposed to trying to get similar scaling capabilities out of throwing
> programmers (or programmer time) at the problem, that's certainly
> "realistic" in my estimation.A lot depends on your application requirements. If you design it from the
ground up to be "shared nothing", then you may well be lucky enough to
truly HAVE shared nothing. But you'll also have a pretty limited feature
set.
"Feature-rich" is overrated. Anyone who tries to be everything to
everyone will end up being not the right thing to pretty much everyone.
You only get into the kind of trouble you describe when you try to hard
to get *everyone* interested.
What's the big buzzword today? Social networking. What did we used to
call that? "Community." What was the single biggest sticky-paper
community feature? Buddy lists. Who does buddy lists besides the Big Guys
(who can throw money at it) and the really small guys (who fit on a single
server)? Nobody. Why? Doesn't scale linearly. Think about what it takes
to offer a feature that, for every simultaneous user, checks the list of
every other simultaneous user for people you know. Shared-nothing *that*.
The answer to that, from where I'm sitting, is to choose between focusing
on "social networking" and focusing on something else. If you're just
adding it as "yet another feature" to your application to become
buzzword-compliant, you'll become another dot-com startup has-been. Of
course, there's also always the business strategy of "look successful,
sell to someone big" without actually turning a positive buck along the
way -- and if that's what you want to do, you're on your own.
My area of expertise was the AOL mail system. And, looking back, there
were a number of core features we offered that simply couldn't be done in a
shared-nothing world over slow phone lines:
<snip a bunch of stuff about features>
And some of the features were only important in an age where pipes (both
last-mile and LAN) were very narrow and disks and RAM were very small.
Spam, in particular, made the "one copy of each message" model obsolete,
because spammers wouldn't play by the rules.But restricting yourself to only shared-nothing features means ruling out
an awful lot of features. Including anything depending on a database
index, or a table that fits completely in memory, or any sort of
rate-limiting or duplicate-detection or spam prevention, or in fact
anything that makes any assumptions at all about the state of any database
you're interacting with or relational integrity or any other transaction in
the system, ever. Including whether the disk drive holding the transaction
you just wrote to disk has disappeared in a puff of head crash.
You don't always have to write shared-nothing code to get near-linear
scalability -- and it's true that near-linear scalability is something
that only exists within certain ranges before you hit a cost or resource
requirement spike, but if you're smart you plan ahead for those kinds of
things. Things don't always go as planned, of course, but if you're
smart you plan for *that*, too, by setting aside "money for a rainy day"
and ensuring that, short of your main datacenters and every off-site
backup in the world being eliminated by meteor strikes simultaneously,
any major scaling issues will not require a sudden "right now" fix.
It was always the little things that bit us. Know why AOL screen names are
often "Jim293852"? Well, it started out as "The name 'Jim' is already
taken. Would you like 'Jim2'?". Guess how well that scales when the first
available Jim is "Jim35000"? Not very.
I'm curious how any of this is meant to support the position that a
faster-executing programming language that imposes greater hurdles on
programmer productivity will be a better investment for scalability than
designing a system that can absorb greater loads by adding hardware
resources.
Pop-quiz: Which of *your* core features would you have to eliminate with
three million simultaneous users?
Hopefully, by the time you have that many users, you're making enough
money to be able to manage that many users. If not, your business model
sucks.
On Thu, Oct 04, 2007 at 07:10:05AM +0900, Jay Levitt wrote:
On Thu, 4 Oct 2007 04:40:52 +0900, Chad Perrin wrote:
--
CCD CopyWrite Chad Perrin [ http://ccd.apotheon.org ]
W. Somerset Maugham: "The ability to quote is a serviceable substitute for
wit."