Global System Services Corporation (GSS) - The leading provider of email and messaging, directory services, and groupware consulting and professional services

Gene,

I suggest looking at http://www.tpc.org. I have developed a related measurement called Messaging Events Per Second (MEPS) for systems like Domino which translates into TPC-Cs, thus if you have a statistical capacity model of the planned Domino system it is possible to find with reasonable accuracy the appropriate hardware requirements. We have done this successfully for a number of clients. We do not use Lotus' NotesBench or Server.Planner.

Published benchmarks are of limited use. Using these numbers requires careful reading because in many cases the numbers are artificial in the sense that they are based on unrealistic tests, e.g., a single protocol or a single application. Ordinary benchmarks are inferior to our methodology which is called Simulation Testing. We've modeled and tested systems scaling to over 1 million users.

Another problem with benchmarks and 'rules of thumb' is that certain variables do not remain constant with system size and complexity. In other words, benchmarks and scalability guidelines are inherently linear but the relationship of system size and complexity to performance and scalability factors is not. At different orders of magnitude server sizing equations can be completely different.

The key to using capacity modeling and simulation testing so that real systems work in the real world is simply understanding, in a quantitative way, how the planned system will be used. Messaging-only numbers, for example, would not be useful for applications and applications vary tremendously in terms of server processor load and network bandwidth utilization, i.e., one application may be radically different from another.

The ideal case is to acquire data about the user population and applications, or from similar systems, and then use this data as the basis of modeling and testing. Our rule of thumb is that 'if you want to do it right, you have to do it yourself'.