>Date: Tue, 11 Jul 1995 09:47:44 -0700
>To:[log in to unmask]
>From:[log in to unmask] (Bruce Toback)
>Subject:Re: Mapped File IO
>
>>Date: Tue, 11 Jul 1995 09:45:26 -0700
>>To:CValdes <[log in to unmask]>
>>From:[log in to unmask] (Bruce Toback)
>>Subject:Re: Mapped File IO
>>
>>>Does anyone know the pros and cons of using Mapped File IO on the 3k's?
>>
>>Some of us know some of them :-).
>>
>>>Is it realy that much faster?
>>
>>It depends on what you're doing. If you're doing random access, it can be a
>>lot faster: by using mapped I/O, you bypass all the "overhead" of the file
>>system and go directly to your data. Mapped access is particularly useful for
>>complex structures like linked lists, B-trees, and so on, where one part of a
>>file references others. In these circumstances, mapped access can result in a
>>performance gain of 5x or more -- in one case, I saw a 12x improvement.
>>
>>On the other hand, sometimes the "overhead" of the file system isn't overhead
>>at all. If you're doing large sequential reads, the file system sees that and
>>prefetches data from the disc, overlapping I/O with your processing without
>>your needing to be aware of it. Under these circumstances, mapped I/O will
>>actually be slower since your process will have to wait while the I/O system
>>fetches the data you reference one page at a time. By working with the file
>>system on sequential reads, you can achieve up to 100% overlap between
>>processing and I/O.
>>
>>You can maximize performance on sequential reads by using a combination of
>>file open options: multi-record, nobuf, and nowait. The multi-record (MR)
>>option tells the file system to ignore file record boundaries and transfer
>>data to your program in as large a chunk as you specify. Nobuf tells the file
>>system not to do its own buffering, but to read data into your buffer
>>instead. Finally, nowait tells the file system to start processing your I/O
>>call and return control to you immediately, without waiting for I/O to be
>>completed.
>>
>>This last option complicates your program somewhat because a call to, say,
>>FREAD, won't necessarily have read anything when it returns to your program.
>>You need to call either IOWAIT or IODONTWAIT to complete or check on the
>>completion of the read call. Obviously the trick here is to set up two (or
>>more) buffers, fill the first, and then start I/O on the second while you
>>process the first. Again, you get greater program complexity but much better
>>resource utilization.
>>
>>Using privileged mode, it is possible to combine this technique with mapped
>>I/O, telling the I/O system to prefetch data from a file while you process
>>the first chunk. I haven't done this myself, so I can't supply any reliable
>>details.
>>
>>>
>>>What should I watch out for?
>>
>>Mapped files can be accessed either "long-mapped" or "short-mapped." The
>>difference is the size of the pointer used to access the data. Short-mapped
>>files are slightly easier to work with since they're accessed with normal,
>>32-bit pointers that you can pass to (some) intrinsics and library routines.
>>On the other hand, you're subject to per-process limits of 6MB maximum
>>short-mapped file space, and no more than 4MB in any one file.
>>
>>Long-mapped files provide access to any size file up to 4GB. Long pointers
>>result in slightly less efficient code, however, and can't be passed to any
>>routines that aren't expecting them. You can't open a file both long-mapped
>>and short-mapped: if one process opens a file long-mapped, all other
>>processes must do the same. In general, I use long-mapped access to
>>application data files, and short-mapped access to small data structures such
>>as interprocess communication buffers.
>>
>>Some file types can't be accessed with memory-mapped I/O, and other types can
>>be opened only with read access. In general, any plain vanilla file can be
>>opened for read access, and any fixed-length record file can be opened for
>>read/write access. Fancier types (spoolfiles, keyed-sequential files, and so
>>on) can't be opened for mapped access by non-privileged processes.
>>
>>One last gotcha: when writing a file with mapped I/O, the file system doesn't
>>automatically extend the file. You need to use FWRITEDIR or FPOINT to
>>actually set the end-of-file. The symptom of failing to do this will be that
>>your program will run normally in all respects, including reading and writing
>>the file, but once the program closes the file, it will have no data in it.
>>
>>-- Bruce Toback
>>OPT, Inc.
>>[log in to unmask]
>>
>