Stan writes:
>Wirt writes:
>> Bill's comments rekindle one of my primary disgruntlements with simple
>> performance analyses. We manufacture a report writer, QueryCalc, in which we
>> do everything we can to drive CPU utilization to 100%. In fact, if we were
to
>> do anything else, we would be derelict in our duty. One hundred per cent CPU
>> utilization is not a sin in a report writer, it is a profound virtue.
>
>I beg to differ...

As do I.

>If your process is using 100% of the CPU, and nearly 100% of the
>theoretical maximum number of I/Os per second, you might at first
>glance say "this is good".  But, let's probe deeper, and try for a shot
>at understanding why the simple rule of "100 % CPU usage is good" doesn't
>hold.
>
>Let's take the above scenario ... we're driving the machine to the max,
>and we're happy.  But ... the *OTHER* users aren't happy, are they?

Other users may be even unhappier than it seems at first blush. The
reason is that while I/O time is elastic, CPU time is not. An HP3000
system capable of 120 disc I/Os per second is capable of doing several
I/Os -- and therefore, serving several users -- in parallel. The CPU can
serve only one user at a time. A multi-user application that is
CPU-limited has a very sharp knee in its performance curve (number of
users vs. response time). A multi-user application that is disc-limited
has a much gentler slope because of the statistical distribution of
resource usage. The CPU-limited system will have a shorter *average*
transaction time, but users care much more about the peak than the
average.

>Tests that one can do to see if the CPU or the I/O is a bottleneck:
>
>   1) prefetch a big test file (but not bigger than, say, 1/2 of
>      physical memory).
>      time your program.
...
>   2) modify the program to use a canned input source, and $NULL as the
>      output.  (Later, try replacing the output writes with empty calls)
>      (i.e., no disk I/O)
>      time your program.
...
>   3) modify your code to not "process" the input records, and time
>      a sample run (still reading the input records, and
>      writing a dummy output record).

Or use a tool like HP's SPT/3000, which does statistical sampling of your
program and determines where it's spending its time. I find that my
extensive education and experience in these matters allows me to predict
likely code bottlenecks accurately about 10% of the time. In one
extensive "panic-optimization" session two years ago (triggered -- sorry,
Wirt -- by 100% cpu utilization and lots of user complaints), my #1 guess
as to where the program was spending its time turned out to be #5 or #6,
and my #2 was in the noise, even after optimizing the first five
statistical hot-spots.

Or removing the hot-spot code entirely. It turned out that the program
was spending almost 30% of its CPU time executing debugging code that had
been left in accidentally. (Unlike some debugging code, this was
*completely useless* in a production environment, since its sole function
was to prepare data structures that could be examined interactively with
the source debugger.) This is something I'd never have found by guessing
or "analysis" since it wasn't supposed to be there in the first place.

Another unguessable bottleneck in this TTY-based inquiry program turned
out to be FWRITEs to the controlling terminals. I discovered that these
take almost constant time, regardless of the amount of data sent, and the
program was doing a lot of them. Better buffering of the output yielded a
further sequential 20% reduction in CPU usage for an average query.
Another sequential 20% was gained by switching from DBGET to priv-mode
data base access (doable because this program does all its DBGETs in mode
4, and has read-only access to the data base). It turns out that TPI
(third-party indexing like Omnidex or Superdex) takes a toll even on
IMAGE calls that don't use TPI features.

In short, I had designed the program with performance in mind,
concentrating on optimizing the complex Boolean searches that users would
be doing. The program did indeed perform much better than the one it
replaced, but when usage increased and CPU time became an issue, it
turned out -- unsurprisingly, in hindsight -- that the performance
problems were in areas that I hadn't considered important when designing
the program. Use of a sampling tool meant that I didn't spend
unproductive time further optimizing search and sort code that was
already performing extremely well.

One other course of action is to build statistics-gathering machinery
into the application from the start. For each query, the program writes a
record to a log file that contains the date and time, the amount of CPU
and elapsed time spent in each key portion of the query processor, the
response time for the query, and a portion of the query text. Examining
these logs enabled us to show management that the new program was in fact
processing two times more queries per hour than the old one, and showed
us that the performance problem we were dealing with was one of peak
response times, not average times. Again, this served to direct our
attention to the right problem immediately.

So designing this particular program according to the principles that
Wirt describes -- optimize I/O so as maximize the time that the CPU is
busy -- gave us a program that did indeed perform better than its
I/O-limited predecessor, but also yielded a program with much less
forgiving overload characteristics. The good news for the client, once
the overload behavior was tamed (see
<http://www.optc.com/Webbot/Thread4642.html>), is that money spent on a
new CPU will have predictable and readily-calculated benefits.

-- Bruce




--------------------------------------------------------------------------
Bruce Toback    Tel: (602) 996-8601| My candle burns at both ends;
OPT, Inc.            (800) 858-4507| It will not last the night;
11801 N. Tatum Blvd. Ste. 142      | But ah, my foes, and oh, my friends -
Phoenix AZ 85028                   | It gives a lovely light.
btoback AT optc.com                |     -- Edna St. Vincent Millay
Unsolicited mail to [log in to unmask] will be inspected for a
fee of US$250. Mailing to said address constitutes agreement to
pay, including collection costs.