Wow,.....I need to work at Allegro. :-)
___
Tony Newton

> -----Original Message-----
> From: Gavin Scott [mailto:[log in to unmask]]
> Sent: Friday, April 20, 2001 10:11 AM
> To: Newton, Tony; [log in to unmask]
> Subject: RE: [HP3000-L] Follow-up to: Searching for the whole CR/LF
> story
>
>
> Tony writes:
> > Well, I received many helpful replies about the file
> transfer problem that
> > one of our consultants was experiencing.  It actually raised more
> > questions for me than just the one we solved.
>
> Something that's true of almost any in-depth inquiry.  When
> you turn over
> rocks, you have to expect there to be a lot of unexpected
> crawly things
> underneath.  Most people of course just sit on the rocks in blissful
> ignorance.
>
> > I also learned (I hope that these are actually correct) that
> >
> > 1) There is no actual MPE/iX EOR marker at the end of a record.  The
> > beginning of the record contains a 2 byte value that tells MPE
> > how long the record is which in turn tells it where to start
> > looking for the next record which contain another value and so-on.
>
> This is true only for MPE Variable Length record files.
>
> For fixed length record files, there is neither a terminator
> character nor a
> length value because the system just knows that every N bytes
> (where N is
> the record length of the file) a new record starts.
>
> For Bytestream files, there are no record separators
> whatsoever and the file
> is just one big stream of bytes that can be thought of as
> just a single
> record.  However there is a strongly enforced convention that
> LF characters
> in the data stream should be interpreted as indicating the
> end of a logical
> record.
>
> "Unix" and "Posix" programs that read "text files" have to
> explicitly look
> for these record separator characters so that they can
> pretend that their
> operating system supports records.  This is a pain when programming in
> things like Perl that leave it up to you to decide when you
> want the LF
> characters and when you want to strip them (which you
> generally have to do
> manually).
>
> Most "MPE" programs on the other hand tend to rely on MPE to
> tell them where
> the records start and end by using the "read a whole record"
> and "write a
> whole record" Intrinsic functions like FREAD and FWRITE.
>
> Since MPE supports both Unix style and MPE style files and
> both types of
> programming environment, there are issues that come up when
> an old style MPE
> program wants to read a bytestream file, or a Posix style
> program wants to
> read an MPE file that does not include the "LF" characters
> that it's looking
> for.
>
> MPE has a set of special "type managers" that get assigned to
> an open file
> that know how to translate back and forth between the two
> ASCII file styles.
>
> When you read an MPE fixed or variable file from a Posix program, the
> Bytestream Emulator inserts LF characters into the data
> stream so that your
> Posix code can still think it's reading a bytestream file.
>
> Then you read a bytestream file from an "MPE" program, the
> Record Oriented
> Bytestream Type Manager will "eat up" the LF characters so
> that your MPE
> program can pretend that it's reading an old style MPE Variable Length
> record file.
>
> Life gets more interesting when you throw things like Samba
> into the mix,
> which may be asked to store PC and Mac style text files in
> addition to the
> "Unix" style of text bytestream files.  While Unix uses the
> "LF" character
> as a record separator, PCs use a "CRLF" pair, and just to be
> different the
> Mac uses a bare "CR".
>
> It would have been nice if MPE's bytestream files had an
> option to specify
> the separator character being used (i.e. the source platform
> that the file
> originated on) which would make interoperability a lot easier.
>
> > 2) Some apps such as Reflections "assist us" by inserting a CRLF
> > at the end of each record when viewing them with "Display
> > Functions" turned on.
>
> How files handle records is a completely different issue than
> how terminals
> control where information is displayed.  The CR/LF characters
> used to move
> the cursor to the start of the next line really ought to be
> thought of as a
> different function than the CR/LF characters in a file that
> indicate record
> boundaries.
>
> The fact that some of the record separators happen to be the
> same characters
> that a terminal can use to make each record appear on a
> separate line is
> more of a confusing coincidence than anything else.  So life becomes
> complicated because:
>
> - A file may have LF, CR, CR/LF, or nothing as "in-band"
> record separators.
>
> - The operating system may insert or remove such characters
> as it tries to
> be helpful.
>
> - The runtime library / language being used to read the file
> may also try to
> be "helpful".
>
> - A program reading a file may or may not end up seeing
> embedded characters.
>
> - A program reading the file may or may not strip such
> characters if they
> are there, or always strip the last one or two characters
> without looking at
> them.
>
> - A program writing a file may explicitly add separator
> characters to the
> output.
>
> - The operating system and or language / runtime may insert
> characters in
> the file output or manipulate those of the user.
>
> - When writing to a device other than a disk file (terminal, etc.) the
> program, language, os, or device driver may insert or
> manipulate separator
> characters or add terminal control characters like CR/LF to
> the output that
> arrives at, say, a terminal or terminal emulator.
>
> - A device such as a terminal emulator may interpret what
> look to it like
> control characters in different ways and might strip some
> characters or
> indicate to the user that some characters came from the host
> when they were
> actually generated by the terminal emulator (CR to go with LF, etc.).
>
> > 3) You can transfer files around in ASCII mode and still
> retain imbedded
> > control characters.  (I guess there must be a difference
> between escape
> > sequences and control codes?)
>
> FTP's ASCII mode relies on each end of the connection to
> convert from the
> local text file record format to a canonical format with explicitly
> indicated record boundaries for transmission over the
> network.  The sender
> strips out any in-band record separators and receiving side
> then takes each
> explicit record and adds whatever local indicator of record breaks is
> required (if any) before writing the output file.
>
> So it depends on the FTP implementation on each end as to
> which characters
> can be transmitted transparently without being interpreted as record
> separators (possibly inappropriately).
>
> Very few things in the world make any distinction between
> "text", "escape
> sequences" and "control characters".  Things like CR, LF,
> NUL, XON, XOFF,
> and a few others are likely to be affected (interpreted as
> some form of
> "control" character) on a non-transparent connection, but most "escape
> sequences" just look like text to everyone except the thing that
> "understands" them.
>
> With the decreased reliance on things like serial
> communication where there
> is little if any out-of-band signally capability, most
> connections have
> become much more transparent to binary data.
>
> G.
>

This message is intended for the sole use of the individual and entity to
whom it is addressed, and may contain information that is privileged,
confidential and exempt from disclosure under applicable law.  If you are
not the intended addressee, nor authorized to receive for the intended
addressee, you are hereby notified that you may not use, copy, disclose or
distribute to anyone the message or any information contained in the
message.  If you have received this message in error, please immediately
advise the sender by reply email and delete the message.  Thank you very
much.

* To join/leave the list, search archives, change list settings, *
* etc., please visit http://raven.utc.edu/archives/hp3000-l.html *