(I've heard that some other answers have been posted while I was writing
this...sorry for any duplication that may occur!)
Re:
> been simpler to implement, document, and debug. As it is, they did an
> adhoc addition to the FOS, as indicated by their implementation design.
> In order to have a valid 64 bit pointer, you must HPFOPEN the file and
> request the 64 bit pointer at that time. Any other method used, even
> with the AIF intrinsics, returns a pointless value.
No!
You're making the same reasonable mistake that I made ... and
others have made.
You're assuming that just because FFILEINFO 64 or AIFFILELGET 4010
is returning a pointer to a file that you can *USE* that pointer!
In point of fact, if the file was not opened specifically for
mapped access, then you can't use that pointer to STORE data!
Basically, the dirty pages of an arbitrary disk file aren't posted to
disk at file close time unless "map_out_must_post" (an attribute of
an open file) is true. When you HPFOPEN (item 18 or item 21 or item
87) a file (i.e, "mapped open"), then MPE says "aha, you're telling
me up front that you intend to use mapped file access to access the
file data...I'll set "map_out_must_most" to true).
IMHO, I think that FFILEINFO 64 and AIFFILELGET 4010 should do:
if the file is opened for in/out or output or append access then
map_out_must_post := true
that change would allow us to do:
FOPEN (or some other kind of non-explicitly-mapped file open)
FFILEINFO (...64...), or AIFFILELGET 4010
use the pointer to store data into the file.
Without such a change, you must use post() to ensure that the changes
you make (if any) are posted to disk.
> By this I mean that copying data, and mixing LARGE_MAPPED with SHORT_MAPPED
> or LONG_MAPPED values is not supported.
sure it is...and is well defined.
If you come along and try to map-open a disk file, MPE says:
what kind of request are you making? (short, long, large)
Is the size of the file compatible with the type of open?
large request: any size ok
long request: up to 4 GB ok
short request: up to 4 MB ok (providing total short space used
by your process will be 6 MB or less)
is the file already mapped open? (if so, how: short, long, large)
large request & already opened short/long/large : OK
long request & already opened short/long : OK
short request & already opened short : OK
> Having done it myself allows me some expertise on the subject. You simply
> abstract the various functionality, and build it on a solid paradigm.
the problem is that the *did* implement the file system that way ...
which is more costly than "flag programming".
I.e., they have a type manager for fixed record files, and one for
variable record files, and one for msg files, etc. This allows higher level
file system code to do some things to files without knowing about the
characteristics of a file (thus, they abstracted things).
The problem with that is that each type manager that wants to support more
than 4 GB must be changed.
While I'm sure you've "done it yourself", I doubt you did it on as
sophisticated an operating system as MPE.
With MPE, for example, consider the work of appending a single record to
a file.
flat file:
locate the EOF.
Is EOF < LIMIT? If not, error.
If record size is odd # of bytes, append a blank (or null).
copy the record data.
blank (or null) fill out to the defined record size.
Increment EOF (which, BTW, is easier than it sounds:
- uses XM to provide safe, recoverable change
- increments EOF in terms of bytes, records, and (IIRC) blocks
variable record file:
Will new record fit in file? (Not the same check as for
a flat file! (I.e., you can have EOF > LIMIT, as long as the
data fits within the calculated worst-case size)
build the variable record information (record size, record data)
copy the record info
(no blank fill)
Increment EOF
In just the above two cases, we have two different methods of determining
if the record will "fit" .. and that code has to be changed to use
64-bit arithmetic instead of unsigned 32-bit arithmetic.
>
> No Stan, I mean on two different machines running the same OS version, with
> the
> same licenses, the same amount of memory, the same software running, the
> same
> CPU Model, the same amount of disc free space(relatively speaking), will
> produce
> different results, in that one will work fine, and the other will produce
> errors,
> so that those sites that are developing 64 Bit code, cannot assume that
> their in house
as in?
The only "inconsistent" result I've ever seen has come from misusing a
pointer returned by FFILEINFO / AIFFILELGET.
> Because of how they implemented Large Files, normal memory mapped access
> does not
> mark a Large File page as "dirty", so writes using a Large Pointer obtained
> by any method
NO...not true at all. There's no difference in making a page dirty
of a Large file, a Long mapped file, a short mapped file ... on 6.5 *or* 1.0!
(Well, we didn't have Large files on 1.0 :)
The page becomes dirty ... as <plug> PAGES </plug> would show.
The problem you're apparently encountering is that the dirty pages aren't
being forced to disk at file close time. Instead, they *may* get
forced to disk by memory pressure .... not something you can count on!
Until HP changes FFILEINFO/AIFFILELGET as I outlined above, you can force
the dirty pages to disk via post(), which I'll be happy to discuss with you.
In summary: this is *NOT* a Large File problem!
> > HPFMOVEDATA ... properly moves data to/from a Large File
>
> This is true, but works only for Large Mapped files, not for SHORT or LONG.
No...works for any data. The calling sequence is weird, I admit, but
it does work. Here's a sample showing it works with any kind of address...
even addresses not associated with a file:
{hpfmoves.source -e hpfmove.pub}
$standard_level 'os_features'$
program m (output);
var
altstatus : integer;
buf1 : packed array [1..32] of char;
buf2 : packed array [1..32] of char;
bytes : longint;
ptr1 : globalanyptr;
ptr2 : globalanyptr;
status : integer;
Procedure hperrmsg; intrinsic;
Procedure hpfmovedata; intrinsic;
begin
buf1 := 'hello there';
buf2 := 'old data';
bytes := sizeof (buf1);
ptr1 := addr (buf1);
ptr2 := addr (buf2);
hpfmovedata (bytes, ptr1, ptr2, status);
if status <> 0 then
begin
writeln ('status = ', status:1);
hperrmsg (2 , 1, , status, , , altstatus);
end
else
writeln ('dest = ' , buf2);
end.
> > HPFMOVEDATALTOR ... ditto, in a known order (allowing "clever"
> > data overlap tricks)
>
> > HPFMOVEDATARTOL ... ditto, in a known order (allowing "clever"
> > data overlap tricks)
>
> These two should have been rolled into HPFMOVEDATA. The software technique
> to do so
> has existed since K&R.
No...it hasn't. There's a darn good reason to have 3 move
routines! Check: man memcpy
and you'll see:
memcpy() Copy n bytes from the object pointed to by s2 into the
object pointed to by s1. If copying takes place
between objects that overlap, the behavior is
undefined. [...]
Note that "undefined"!
If you want that behaviour, use HPFMOVEDATA.
If not, consider using HPMOVEFDATALTOR or ...RTOL ... each has its place,
because without having a separate routine, a "common" (merged) routine
(without an extra parameter) would have no way of knowing what your *goal*
is, and could guess wrong.
> >On a 927, HPFADDTOPOINTER takes about 24 microseconds.
>
> Assume that you are processing a file and you have a choice between using
Go ahead...roll your own. I did.
HP wrote theirs the way they did for several reasons:
1) callable from all languages ... even languages that don't have
the concept of a pointer or a 64-bit integer!
2) tests that the address of status and output_ptr are legal
addresses ... necessary because they're coding to the intrinsics
standard (which says: thou shalt check the validity of all
addresses, so that (a) you don't abort; and (b) you don't allow the
user to corrupt memory).
That said, I'd have done it differently:
1) if the source pointer is nil, adding anything to it should be an error
(but it isn't)
2) if the source + offset > $ffffffff.ffffffff then it should be an error
(but it isn't)
3) I'd provide a quick version, but ... the intrinsic mechanism
doesn't like 8-byte *value* parameters (other than 8-byte floating
point numbers). (thanks, COBOL :(
> >> The calls to
> >> get a 64 bit file pointer to a LARGE file that is already opened, cannot
> >> be used, even though they are added with 6.5 and documented, even though
> >> they appear to work, almost all the time, and on some machines, 100% of
> the
> >> time.
>
> >Huh?
>
> AIFFILELGET, 4101.
No...that's not an explanation.
In general, you're too often saying things painting things with too
broad a brush. You *NEED* to discuss problems ... that's how
you find out when you're wrong, and how we find out when MPE's wrong
... and both things happen!
> >> The documentation for the new intrinsics they provided are truly at
> >> the most fundamental level, incorrect(Parameters are not the types
> >> described), and incomplete.
>
> >I noticed one small difference between the *preliminary* documentation
> >I received, and the actual final documentation.
>
> ><snip></snip>
>
> The problem is that one should pass a SHORT pointer to the LARGE_POINTER,
> instead
> of what is described in the manual.
????
HPFADDTOPOINTER, the intrinsic I was thinking of, takes long addresses
pointing to long addresses. Similarly, the three HPFMOVE... intrinsics
take long addresses pointing to long addresses. I.e., the only short addresses
I can see are for the "status" parameters.
This is corroborated by testing on an unpatched 6.5 release just now.
(The change I was thinking of was that the early version of HPFADDTOPOINTER
was a function, not a procedure.)
> >> and their refusal to do so is further evidence. That management allowed
> a
> >> design and implementation that consistently, knowingly, produces
> >> inconsistent
> >> results, would in some cultures result in such "loss of face" as to have
> the
> >> problem be "self remediating".
>
> >I'd be interested in knowing more about this. Are there bugs in
> >Large Files? (I know of one, which has a patch, which affected the
> >reporting of the EOF in some cases.)
>
> It is not so much of a "bug" as it is that if you read the documentation as
> a normal
> person might, and then implement something, it may work perfectly for you.
> It is
> only when you start hammering away at the LARGE_MAPPED section of the code
> that you
> might - just maybe - discover a problem.
Such as?
You're NOT being helpful, nor fair. Simply saying "you might discover a
problem" provides no useful information. Many "problems" that I've heard
users complain about with MPE have turned out to be user misunderstanding.
This has happened to me, as well. Without information ...
preferably well documented ... we can't judge what category your
problem(s) fall into!
> The other problem is what you have to go through to support LARGE files. In
> the case
> of a single application doing a single task, it is a no brainer. If you
> have to
> call their intrinsic to add a constant to a pointer a dozen or so times in a
> single
> IO bound loop, no big deal. In the case of someone trying to do something
> generic,
> and applicable across all cases, the tools HP has given us, are inadequate
> to say the
> least, and show the level of commitment to the MPE which was my original
> point.
Extremely few programs use mapped file access ... those that do are
generally written by people who can evaluate the cost of calling
an intrinsic to add to a pointer vs. writing their own routine.
...remember, some of those programs are written in COBOL (yes...COBOL),
FORTRAN, and other languages. The intrinsics HP provided are a means of
solving a problem across all programming languages.
//
Stan Sieler [log in to unmask]
www.allegro.com/sieler/wanted/index.html www.allegro.com/sieler
|
