I thought I'd mention that the "enhanced checkpoint" feature
(ALTERCHKPTSTAT in VOLUTIL) is not used as of 6.5, the command
remains in volutil (no idea why!) but it does nothing. The feature used a bit
 map to track changed pages and that didn't scale with large files. And the
"system-wide" semaphore mentioned is also gone replaced with a more
granular object based locking scheme.

And I thought I'd also share some historical information about the early days of 6.5
and 7.0 with large memory and large files that might help put the "scanning memory"
statement into perspective. Some of this might have been discussed here several years
ago.

The memory manager scans memory. When XM needs to ensure that pages of
files have posted to disk (been made durable) so that it can reuse a log half it calls
memory management (MM) routines to do that.

In addition to handling  post requests or fetch (or prefetch) requests MM has
to try to keep memory organized. These activities include making present pages
into "recoverable overlay candidates", or "roc" pages if they have not been accessed
recently. This may also include starting a background write if the page is dirty. MM
will also try to take pages from the roc list and make them absent (free) if  they
remain unaccessed and if their background write has finished. This activity will
become more urgent as the pool of free pages drops below certain thresholds and
can include bumping the priority of background writes so they complete more
quickly and being more aggressive about roc'ing present pages.

These list management algorithms scale based on memory size. And unlike on
MPE V where this activity could occur during idle periods because there was far
less memory to manage, on MPE/iX we don't have that luxury. The memory
manager has to try to do some amount of  list maintenance almost any time
it is called.

Early in 6.5 on systems with large memory and large files we found that these algorithms
did not scale as well as we would have liked. They might take too long by trying to
do too much, too frequently. And this may be where the "scanning memory" observation
was made.

We made a number of  enhancements to speed memory management activtities. This
was **several years ago** and by now I'd hope most 6.5 and 7.0 systems have these
patches. Here are some of the more significant of those large memory performance
improvement patches:

MPELXG6 - The first of two enhancements to the memory manager list maintenance
algorithms. This reduced the length of time MM would spend traversing its lists and
while doing so, keeping parts of memory locked.

MPELXH8 - The second of two, further shortening the amount of time memory
manager locks are held and changing the frequency and location of some of the
free page replenishment activities.

MPELXF8 - Enhancement to storage management allowing "big" files of 1GB
or more to be held on a "least recently used" list longer. This list holds "GUFD's" or
"global unique file descriptors" of files that are closed and have NO accessors.
The expectation being that normal memory management activity might whittle away
at the file pages posting them and minimizing the impact of the post that would have
to occur when the "GUFD" structure needed to be reused for another file.

MPELXH5 - The "whittling" wasn't happening fast enough so we added code to
do that. It's done in small increments from the end of the file upwards so if the file
reopens before it is fully mapped out we minimize the page faults needed to bring it
back into memory.

MPELXF2 - An enhancement to a memory manager internal api called make_pages_roc
that allowed callers to make pages free rather than just recoverable. Rather than letting
the memory manager discover these unneeded pages, it can be told that the pages are no
longer needed and can be tossed out of memory right away.

MPELXJ9 - A further improvment to that enhanced api so both present and recoverable
pages would be made free. The initial code change in LXF2 unnecessarily skipped roc
pages.

These patches are all superseded by others now so for 6.5 patch MPEMXE5A
will install all these changes and many more. The 7.0 patch is MPEMXC7A. These
changes were submitted to 7.5 so there are no 7.5 versions needed.

This is probably much longer than it needed to be but I hope it helps.

Cheers,

Bill
hp/vCSY
===========================================
  Reply to: bill . vcsy -at- comcast . net
===========================================

* To join/leave the list, search archives, change list settings, *
* etc., please visit http://raven.utc.edu/archives/hp3000-l.html *