On Mon, 18 Jun 2001 14:34:23 -0700, Steve Dirickson (Volt)

>> [discussion: peek / process / destroy data in message files]
>> [attested to be "the right way" to do this ;) ]
>
>I've seen this method advertised as the "right" way to handle message
>files a number of times over the years, and I've never bought it, for
>several reasons:

Hmmm... as they say, there is more than one way to flay feline flesh, so I
won't (completely) argue with you on these points, just point out my view...

> 1) The usual reason for using message files is for speed: to quickly
>transmit some information from one activity to another. That's why
>message files are memory-resident and aren't flushed to disk. Doubling
>the reads (and throwing in otherwise-unnecessary FCONTROL calls) kind of
>kicks the speed boost in the face.

The FCONTROL(6) calls [to "flush" the buffer] are, as you say, somewhat
excessive and potentially costly if "speed" is the issue, however I doubt
the second FREAD is really all that "expensive" -- as you point out, the
record is already in memory, so the net effect of this second FREAD is to
move the file pointer to the next readable record.  [the memory-to-memory
transfer of the buffered data should take nanoseconds at most...] If there
were an FCONTROL or other FREAD option to indicate all that is needed is to
update the file pointer to the next available record, this "expensive"
FREAD would go away in cases where the processing time was so great the
already-read record cache had been reclaimed by the system, thus forcing
another physical transfer-from-disk.

[aside: message files are not necessarilly "memory resident" as you imply --
 it's just that in a typical processing environment, the "reading" process
reads the data before the buffers are posted to the disk, so it appears to
be "memory resident" -- if you fill a 1gb message file faster than you can
process it, I'll bet not much of it is "in memory" on a 256k machine...]

> 2) If the data being passed in the message file is so critical that you
>can't afford to lose a message in the event of a system crash (or your
>application fails so frequently that it's an issue), you might need to
>re-evaluate your use of message files (or get a better app that doesn't
>blow up so much). Critical, can't-lose-it data belongs somewhere safer,
>like in a database.

Application failures won't create a system crash record [unless it clobbers
the entire box :( ]  Sure, a database will have a better chance of not
losing the data, but that adds to the programmer burden [i.e., what to do
when the dataset fills up "temporarily" because the process reading-and-
deleting records is running behind...]  And, while it's "doable" to read &
write to "remote" databases, I *think* message files are optimized for this
case (system-to-system).

> 3) Double-reading breaks completely in any multiple-reader scenario.
>Unless the message is accessed exactly once by a destructive read, it is
>likely to be "processed" by any number of the readers--all but one of
>whom will then destructively read and throw away a valid, unprocessed
>message in the name of "clearing" the just-processed message.

multiple-reader breaks just about every "data integrity" scheme you can
come up with -- if I were using a database, how would I guarantee that
process "A" doesn't DBDELETE the record process "B" is working on? [quick
answer, DBLOCKing around the transaction, but that then single-threads the
application and doesn't allow "multiple" readers to process
concurrently...] [not so quick followup: if the DBLOCK is so constrained
that it only "locks" one logical record, then perhaps a second
DBLOCK/GET/DELETE can overlap, but again, complexity increases & speed
decreases...]

Fortunately (in my case) my application is multiple writers to a single
reader (with multiple "return path" message files so the single-reading
process can route replies back to the correct recipient...)

> 4) Even with a single reader it is subject to duplicate processing in
>the event of a system or application failure: the app reads the message
>non-destructively, processes it, blows up, restarts, and processes the
>message again.
>
>The last issue is probably at least part of the reason for the FSERR 151
>in the first place: to notify the application that the message it just
>read--which is a PERFECTLY GOOD message--was the last message written
>before a processing break, and may need to be handled specially.

Agreed -- unless there is some mechanism to allow the "processing" process
to recognize an already-processed message, this will be a problem for
peek/process/destroy processing.

* To join/leave the list, search archives, change list settings, *
* etc., please visit http://raven.utc.edu/archives/hp3000-l.html *