Second try...

Steve quotes somebody:
> > error, then it means the program is doing the right thing
> > when dealing with message files: issuing a non-destructive
> > read, processing the record, then issuing a destructive
> > read to clear the record.

This can be good *if* the rest of your code is set up so that if the program
fails that the record isn't processed twice, and if you're trying to protect
against system failures then things get a lot more complex since you may
wait until after doing a DBPUT to delete the record from the message file
but after a system failure in the middle of this it's possible for neither
the DBPUT nor the message file record to be there after rebooting, since
unfortunately even NM message files can't be attached to the transaction
manager :-(

> I've seen this method advertised as the "right" way to handle message
> files a number of times over the years, and I've never bought it, for
> several reasons:
>  1) The usual reason for using message files is for speed: to quickly
> transmit some information from one activity to another. That's why
> message files are memory-resident and aren't flushed to disk. Doubling
> the reads (and throwing in otherwise-unnecessary FCONTROL calls) kind of
> kicks the speed boost in the face.

Well, speedy interprocess communication is *one* of the things you can use
message files for, but it's by no means the only one.  "message files are
memory resident" is a common misconception.  Most message files have their
data written to disk just like ordinary flat files.  In fact it's this
persistence of message file contents that makes them useable in a
store-and-forward fashion in addition to a real-time inter-process 'pipe'
arrangement.

As far as memory residence goes, there was an obscure feature of message
files under MPE/V (and I have no idea if it works the same way on MPE/iX or
after the conversion to Native Mode a few releases ago) which is that if the
record size and limit of the message file allowed all of the blocks of the
file to fit in the "buffers" requested for the file, then no I/O would be
done to disk during the reading and writing of the file.

So if you had a tiny message file with a big blocking factor and opened it
with, say, :FILE mymsg;BUF=16, then you could get the behavior that Steve
describes.  Once the limit on the file exceeds the space in the "buffers"
then the message file implementation reverted to a different mechanism of
managing the message file blocks which worked much like an ordinary disk
file.

AFAIK, message files today are not any more memory resident than any other
type of file.

>  2) If the data being passed in the message file is so critical that you
> can't afford to lose a message in the event of a system crash (or your
> application fails so frequently that it's an issue), you might need to
> re-evaluate your use of message files (or get a better app that doesn't
> blow up so much).

The problem is that designing a "can't afford to lose a message" system has
lots of complications beyond just message file limitations.  The only
serious problem with message files is that you can't attach them to the
transaction manager.  If you *could* then you could use the read
non-destructively, update database, read destructively sequence and be sure
that after a failure the worst case would be that the record still exists in
the message file even through it has already been processed once (which the
program can hopefully be taught to look for).

But since message files can't be attached to XM, you easily can't control
when your message file changes actually get committed relative to your
database updates.

Unfortunately most applications written for the 3000 can't survive system
failures without some confusion (duplicate or lost transactions and
synchronization problems between separate files and databases).

>  3) Double-reading breaks completely in any multiple-reader scenario.
> Unless the message is accessed exactly once by a destructive read, it is
> likely to be "processed" by any number of the readers--all but one of
> whom will then destructively read and throw away a valid, unprocessed
> message in the name of "clearing" the just-processed message.

Yes, but the ability to have multiple readers and the ability to do
nondestructive reads are to orthogonal features of message files.  As the
"Swiss army knife" of the MPE file system, you don't want to leave the main
blade sticking out while you're trying to use the can opener.

>  4) Even with a single reader it is subject to duplicate processing in
> the event of a system or application failure: the app reads the message
> non-destructively, processes it, blows up, restarts, and processes the
> message again.

The problem of preventing duplicate processing of a record can't be
prevented as long as you have multiple entities which cannot be covered by a
single "transaction".  In these cases the best you can do is make sure that
you've committed the output before you delete the input, at which point a
failure leaves you again with an input record that has been already
processed.  Because of the no-XM limitation of message files, you can't even
get to this level of security in the presence of frequent system aborts (you
can make the windows of opportunity for problems relatively small however)

> The last issue is probably at least part of the reason for the FSERR 151
> in the first place: to notify the application that the message it just
> read--which is a PERFECTLY GOOD message--was the last message written
> before a processing break, and may need to be handled specially.

Yep, and this is a pretty nifty feature if you find a use for it.  Message
files can contain data, act as persistent queue files, multiplex and
demultiplex data between multiple readers and writers, serve as a process
interrupt mechanism, and provide guaranteed notification of the termination
of other processes, among other things.  They are one of the gems of the MPE
system, even with the possible data integrity issues surrounding system
aborts (which not all relevant to all of their uses).

G.

* To join/leave the list, search archives, change list settings, *
* etc., please visit http://raven.utc.edu/archives/hp3000-l.html *