ESCAPE VELOCITY

1 - the point was that you by adding more memory could remove IO load on the storage subsystem.

8gb memory modules are still 500+usd but stacking 12*4gb in a ML370 is a cheap option but going to 8gb kills most budgets.

2 - Fleming, that doesn't explain anything. They specifically mention "doesn't RAM cost money?" So when they say it's cheaper than disk, surely they're talking about currency, not I/O.

Otherwise why even mention the "up to 32GB?" Is RAM over 32GB suddenly more I/O expensive than a hard drive? That's not true either.

3 - -2 - Fleming, that doesn't explain anything. They specifically mention "doesn't RAM cost money?" So when they say it's cheaper than disk, surely they're talking about currency, not I/O.

in the example they reduced the IO load from 1.0 to 0.3 I/O's with going from 4 to 24 gb of memory , so they would need a storage system that could handle 1500 I/O's vs 4000 so memory is a cheaper solution than buying more spindles.

as for the sweet spot being at 32 i dont know if its due to scaling or the numbers they used at that time.

4 - I can say from personal experience that he is way off base. In 2006 I implemented an IBM BladeCenter with 5 blades along with an IBM DS4300 SAN that had 14 fibre channel hard drives. I had spent the better part of a year getting the specification finalized, so I had seen hardware prices change for a while.

The FC drives I bought were just about the most expensive available. Each 72GB drive cost over $800 ($11.50/GB). Adding the SAN and RAID overhead pushed the per-gig cost to around $16.

When pricing the server blades the single biggest expense was the RAM, and we were only loading 8GB RAM in each one. I don't remember what the memory specifically cost, but I do remember that using 4GB modules was more than twice the cost of using 2GB. And I distinctly remember 1GB modules were just under $100 apiece.

Extrapolating from there it's obvious that disks were only a fraction of the cost of memory in 2006, the same as it has always been. I'm not sure why he mentions specifically capacities up to 32GB, because I can't make the math work no matter what.

5 - Another victim of the public edyoucashun system? { Link }

Seriously though, you are correct. Disk is cheap, even at the SAN level. RAM prices always fluctuate but essentially are ALWAYS more expensive on a gb/gb comparison than disk, pricewise.

I can see Flemming's point though, if Chris Mitchell is a performance tech, than he possibly could have meant that it is more expensive to go out to disk than it is to read from RAM over disk versus the actual dollar cost. That certainly falls in line with "server performance geek lingo".

6 - @3 - That still doesn't make any sense. How does the fact that they reduced disk I/O make RAM cheaper per GB than disk?

It might make adding RAM a cheaper scalability approach than adding servers. But that didn't make RAM cheaper than disk. It just means that RAM is cheaper than entirely new server systems.

I'm wondering if this is just a language barrier.

7 - @5 - But he says "RAM costs money!" That's some pretty twisted pretzel lingo if he wants to talk money in one sentence and then in the very next, use "cheaper" to mean performance overhead.

8 - -How does the fact that they reduced disk I/O make RAM cheaper per GB than disk?

in the TCO of things it will wording might but a bit odd but in the full context at least it made sense to me.

Use X on memory and save XX on storage would be a easier way to put it.

9 - @8 - You don't save a dime on storage. You still have to write all that data to the drive EVENTUALLY. Otherwise why ever hook an Exchange server to a drive system at all? Power goes out -- no problem! You just cleared your Inbox!

If there's 2TB of mail, you'll need a disk system that supports at least 2TB.

What you might mean is: it's cheaper to buy that RAM than it would be to implement a disk subsystem that the OS treated as a paging file to get the equivalent available memory. But even that isn't true. It's very inexpensive to throw a 64GB spindle on the machine and tell the OS "this is virtual memory."

It'll be slow as hell, but it'll be cheap.

RAM is not, and never has been, cheaper than disk in dollar terms.

10 - - You don't save a dime on storage.

you can but a slower storage system on the expence of memory , if you only need to buy a storage system with a 1/3 of the IO's it will be alot cheaper.

11 - -What you might mean is: it's cheaper to buy that RAM than it would be to implement a disk subsystem that the OS treated as a paging file to get the equivalent available memory.

Nope.

Example from the post if you have 4000 mailbox users you will at 24gb of memory use a 1/3 IOps pr mailbox if you used 4gb of memory you would need 1 IOps pr mailbox needing more spindles to do the same job.

/night eurotrash time mere will look here tomorrow.

12 - Yep it's definitely twisted. Let's try this theory and see if it floats:

"Maybe he means that even though RAM is expensive, it's money well spent for performance."

The 32gb of RAM part doesn't make sense other than he might be quoting it because that is the testbed stuff he was working on at the time.

I be MS sExchange 2007 would run great on an ioDrive with minimal RAM. { Link }

13 - @11 - You don't need any more spindles to store the mail. You need more spindles to get the *equivalent workload per spindle.*

Well, so what? I mean, obviously that's good for performance. But so would loading the entire database into a RAM drive and just running everything from there (which is just the logical conclusion of their performance strategy.) That didn't make RAM cheaper than hard drives.

I'm really not even sure if it's a cheaper route to equivalent workloads per spindle. If HD is 1/3 the price of RAM, it's a wash. You'd just have a data center with a bunch of racks filled with small drives that are mostly-empty.

(Okay, I'll grant that floor space, power and cooling requirements might move the line a bit, but in 2006, the price differential was around 5 or 6x per GB between drives and RAM.)

14 - Also speaking from experience, you want every spindle you can get to spread Exchange's workload as wide as possible. Reducing spindle count for mailboxes is definitely not a good idea, so there is no way that is what the author was advocating.

15 - In the context of the blog post I read it too mean that to achieve better performance it was cheaper to add RAM than to speed up the disk sub system. Nowhere does it say that a GB of ram is cheaper than a GB of disk. Given that the whole post is about performance I don't think it's a stretch to assume that cost:performance is the comparison being made, not cost:volume.

Of course it may still be nonsense, but not to the degree you assert.

16 - @15 - WTF? What part of "doesn't RAM cost money? Yes it does, but it's much cheaper than disk up to about 32 GB" is ambiguous!??!?!

"Cheaper to add RAM than to speed up the disk subsystem?" Who can argue with that? It's cheaper to add a disk subsystem than to transcribe the ones and zeros to stone tablets, too.

Why don't we just say "it's cheaper to migrate to SSD storage arrays than to migrate to a new version of Exchange?"

17 - I'm the Chris Mitchell that write the referenced post. The comment was based upon RAM reducing the IOPs needed from the Disk Drives. At the time, a SAN disk drive (we'll assume a 10K FC drive was raw street price around $450 and from SAN manufacturer with warranty close to $1000 and there is additional cost to cabnents and additional) compared to RAM at $100/GB. Also (not in the post), 1GB of RAM roughly saved 100 IOPs...so RAM was cheaper than another disk.

The conversation was important at the time because we were asking admins to move from buying 4GB (max in 32-bit systems) to buying 16-32GB.

For Microsoft IT ({ Link } these improvements along with buying appropriate RAM allowed for the purchase of much simpler and cheaper DAS systems that not only saved substantial money and allowed 2GB quoatas, but improved availability and descreased operations complexity.

Hope that explains it...

18 - @17 - Thanks for stopping by and offering some clarification, Chris. So your argument that RAM is cheaper is in part based on the idea of moving ALL Exchange storage from SAN to DAS. Okay, that's starting to make some sense.

The statement "RAM is cheaper than disk" really required a lot more detail. Since you were clearly talking that it was FISCALLY cheaper than disk (rather than computationally cheaper), then there has to be a unit of comparison. Typically, when one is talking about processing mediums that are measured in byte-size, the comparison would be by byte-size, right? Talking about $/GB is a very typical way to discuss such comparisons.

That's obviously not what you were talking about.

You say that "1GB of RAM saved 100 IOPs... so RAM was cheaper than disk." We must have a language barrier here, because while I understand the idea that I might have to add more drives to support more users, I don't see why that's a multiplicative relationship. Are you saying that to support 1000 Exchange 2003 users with 2GB quotas, I need more than 2TB of storage? How much would I need? Several replies here suggest that prior versions of Exchange favored more individual disks, so I'd imagine that a larger number of smaller drives would yield better overall results due to seek times on those drives. That makes sense, but the efficiency of those drives would have to be very low indeed to increase their cost to the prices of RAM.

Now, the link you point to says that CCR allowed MSFT IT to shift from SAN-based storage to direct-attached storage, which is definitely cheaper per GB. That means that if I'm supporting 1000 users on my Exchange cluster, I can move from a shared disk I/O system to individual direct disk systems for those servers. That now increases the total capacity of the actual drives I require, but makes them less expensive at the per GB margin. I can see where that would save money.

But that doesn't make RAM cheaper than disk. That makes a 1000-user server for Exchange 2007 cheaper than a 1000-user server for Exchange 2003. Assuming the numbers really do add up that way, at least. (And I'm willing to concede that they do.)

You have to completely reconfigure the way you manage drives, though. Nearly every enterprise shop that I know of likes SANs because they're also easier to back up than DAS. So even if your purchase price per GB of storage drops, your TCO/GB *may* go up.

And since your method of comparison is really "cheaper per supported user," I can't really regard that is irrelevant.

Look, please don't misunderstand -- even though I'm a lifelong advocate of your competitor's technology -- Exchange 2007 is clearly a dramatic improvement over Exchange 2003. Proper leveraging of 64-bit platforms is an excellent advancement, and a notable engineering feat. I applaud the team's success in delivering that.

But simply saying "RAM is cheaper than disk" still seems like a disingenuous claim.

All that being said, I must confess that I celebrate the fact that my preferred messaging platform isn't dependent on RAM capacity to determine what my user storage quota is. And that's a good thing, because quotas are nothing more than IT's way to transfer their costs back to the business. Every moment that a business user is managing email to meet a quota is a moment that they aren't doing their actual job.

All of our users would be better served by the term "quota" being scorched from our lexicons.

19 - where is Meta's pic

20 - @17 "4GB (max in 32-bit systems)"

A PC with a 32-bit operating system cannot use more than 3.25Gb of memory due to MMIO