Dale asks:

> I am developing a client-server based application that requires a lot of
> data transfer between the two program components. On the server side,
> there is a program which accepts the data buffer from the client and
> hands it over to some other process running on the same machine.  In
> order to facilitate interprocess communication of the data, we are
> currently using short-mapped file mechanism.


> Unfortunately, with that mechanism, MPE limits the amount of data
> transfer to be no greater than 4MB. Since we need to transfer more data

I have to ask:

   Are you transferring more than 4 MB in a SINGLE CALL?

   If yes, then I'd look into passing the data in a flat file.
   The file could be long mapped (using a couple of simple access
   routines to get around the C shortcomings), or it could be
   accessed via file system.   But, either way, that's going to be
   the fastest approach for passing more than 4 MB in a single call.

   If no, then you have to ask:

      What throttling mechanism exists?
   Or:
      What prevents the creator of the data from outpacing the
      receiver?
   Or:
      How much un-read data can be buffered up before an error occurs?

If the receiver consumes the data faster than the supplier can
generate it, then using a short-mapped file should suffice ...
you'll simply have to either use a single message at a time
or develop a buffer-swap technique (e.g., fill one 2 MB "buffer"
while the receiver is extracting from the other 2 MB "buffer").


I find that in many cases, using message files works well ...
they provide a natural overflow buffering mechanism (assuming the
LIMIT is large enough), and good speed (assuming the receiver
is fast enough).

However, without question, the highest speed alternative will always
be some mechanism using a shared virtual address.  This is because
you can avoid the overhead of copying the data once.

Given that, the question becomes one of:

   32 bit or 64 bit pointers?

In MPE, without resorting to privileged mode choices, shared virtual
memory basically means mapped files.

As you pointed out, the 32-bit pointer choice limits you to
a maximum of a single 4 MB short-mapped file.  You could also
choose to have two 3 MB short-mapped files instead, if you wish.
(The limit is 4 MB in a single file, and a max of 6 MB per process)

A number of C programmers are happily using long-mapped files.
They usually create a few access routines, often in
SPLash! or Pascal, to use the pointers.

E.g.:

    move_64_to_32 (char ^source, char *dest, int32 bytes);
and
    move_32_to_64 (char *source, char ^dest, int32 bytes);
and
    move_64_to_64 (char ^source, char ^dest, int32 bytes);

The main drawback of using mapped files is usually the need to
invent an interlock mechanism of your own.  For non-priv use, I
recommend using an empty file that you can FLOCK as desired.
(Although it's tempting to use the mapped file as the lock
file, that's expensive)

--
Stan Sieler                                          [log in to unmask]
                                     http://www.allegro.com/sieler.html