Ken writes:
> That a value returned cannot be used is clearly erroneous.

First of all, as Jerry pointed out, this is not a large files issue at all,
as MPE has always behaved in this fashion.

There is nothing flaky or intermittent about the behavior once you
understand it.

I believe that you are using the AIFs to ask the system for the virtual
address of an open file, and then proceed to modify the file using that
pointer, right?  The result being that your changes don't always end up
being permanent depending on system load, phase of moon, time until next
system restart, etc.

The problem is that you're asking for the address of the file, but you
haven't told the system that you intend to *modify* the file using that
pointer.

Mapped access (using the pointer to access the file contents) bypasses the
MPE file system and relies on the memory manager to page in bits of the file
when you need them.  There is no automatic mechanism in the memory manager
to ensure that at some point all of your changes to this particular range of
the Virtual Address Space get posted to disk as a group.

When you HPFOPEN a file and specify one of the "mapped access" options, you
tell the file system that you intend to operate on this *file* using mapped
access.  When you fail to do this but later get the address of the file via
the AIFs (behind the file system's back) then the file system has no
responsibility for your mapped access.

When you specify at HPFOPEN time that you intend to do mapped access to a
file, the file system will, at the time you close the file, cause the
Virtual Space Management system to walk the virtual address range of the
file and post all "dirty" pages that it finds which you have changed via
mapped access.

Without the HPFOPEN time mapped access option, the file system assumes that
you have performed all operations using its facilities, or that if you have
gone behind its back to modify the file that you have also arranged to have
your changes posted as necessary.  Since the file system thus skips the step
of scanning for pages which might have "somehow" become dirty, and the
memory manager just thinks about your file as yet another range of memory
addresses, then any changes you've made via mapped access may not make it to
disk for some time / ever depending on the load on the machine, etc.

Yes, this needs to be better documented, and yes, you're certainly not the
first one to run into the problem, but if you start telling the file system
at HPFOPEN time that you will be doing mapped access, then all of your
problems in this area should go away.

Of course mapped access is generally the *slowest* way to access a file,
which is another reason why it's not very well documented since almost
nobody uses mapped access.  Lack of support for mapped files in NetBase was
never a significant issue, which indicates that few, if any, applications
used it for access to actual user data files.

As far as the C compiler support for hybrid long-pointer segmented address
arithmetic goes, I suspect it was felt that the large cost to enhance all
the compilers to support this new data type was not justified due to the
fact that very few people use mapped access, that such fundamental compiler
changes would likely result in new bugs that would be difficult and
expensive to track down and fix, and that it would further "fork" the
compiler development work between MPE and HP-UX, since the HP-UX people did
not want this stuff cluttering up their compilers and their fancy
optimization stuff.

Cheers,

Gavin