Lars writes:

> Stan wrote...
>
>  >Also, a notify mechanism is always more timely than a polling mechanism,
>  >and doesn't require fine tuning (e.g., how frequently the poll happens).
>
>  With smbclient the 3000 might drop the file on the PC disk (into a shared
>  directory) and help putting the "polling burden" on the PC side ;-) ;-)
>  To address the issue of "seeing the file too early", one might put into
>  a separate directory and then rename to the polled directory.
>
>  Lars (still not a fan of polling approaches)

There's nothing intrinsically more unreliable about polling processes than
notification processes. Indeed, it's possible to argue that polling is
substantially more reliable than notification.

Good engineering design must serve at least three purposes simultaneously.
The design, regardless of whether it is in hardware or software, analog or
digital, must be robust, reliable and resilient if it is to be of any value.
At the same time, it must also very easy for the customer to install and use.
Thirdly, the design must intrinsically minimize support calls, which after a
fashion, is simply restating the first two design criteria.

Polling is a far simpler and much more generally applicable design
architecture than any form of notification scheme. Both methods can fail due
to either race conflicts or very busy remote servers, but both can also be
designed to be very robust and resilient.

In that regard, the most common notification scheme that most people are
familiar with is the XON/XOFF process that serial printers used to notify
their hosts when their buffers were filling and the printers needed their
hosts to stop sending output for a period of time. However, it took very
little time to discover that if the host computer was busy, it had a tendency
not to see the XOFF or XON commands that the printer was sending. If the host
missed the XOFF command, data would be lost as the printer's buffer
overflowed. If it missed the XON command, printing simply hung. HP's solution
was to invent "Robust XON/XOFF" and put this protocol in their printers. If
the notification signal wasn't acted upon in a certain amount of time, the
command was resent, episodically repeated, until the desired action occurred.
However, as you can imagine, any form of notification scheme can get very
complex very quickly -- and in the case of file transfers, you first have to
design a communications protocol that isn't currently even present.

Polling, on the other hand, is very simple to implement and if implemented
well, is virtually failure-proof. The scheme that Lars suggests, using SAMBA,
won't do much to increase overall reliability. The far easier way to do it is
to:

     1. Create the file to be downloaded, using a query language, in some
foreign directory/group.

     2. Once done, COPY (filecopy) the source file into the watched
folder/group. This is generally a very high-speed process that will run at
the speed of the host's backplane.

     3. Serve the directory list of the watched folder to the remote polling
computer when asked.

     4. Once the directory is received at the remote computer, have the
remote computer sit quiet for 5 or 10 seconds to absolutely insure that the
directory entry does not currently point to an entry for which no physical
file yet exists (this 5 or 10 seconds wait period is excessively long when
compared to the processes they're waiting on).

     5. Commence the download. If the FTP process attempts to download a file
that is still being written to by an exclusive writer, FTP will fail. If it
does, nothing catastrophic occurs. The remote polling computer will simply
come back around to this server in 10, 20 or 30 minutes, at whatever interval
it's polling frequency has been programmed to be and pick the file up then.

     6. On the other hand, if the FTP can gain control of the file, the file
will first be downloaded and then purged from host server. At that point, the
purpose of the polling has been served.

This simple process is resistant to the server's busy level. A very busy
server will be as slow to process FTP requests as it will be to copy files.
Even more importantly, in any form of race conflict, it is important to note
that FTP will be a dramatically slower process than internal file copies, and
that only works in favor of the polling process not being exceptionally prone
to failure.

We've run a large number of tests against MPE using the general scheme above.
The only failure mode that we've been able to induce using tests to create
maximally likely conditions of conflict is one where the FTP process manages
to get it's transfer request in during that very tiny window when the new
file's name has just been entered into the directory but writing has not yet
begun into the new file. At that point, there is no exclusive writer process
holding off other openers, thus we "transfer" down a zero-record file and
then mistakenly delete the file from the server.

Because this window for this failure is so tiny, no matter how busy the
server seems to be, simply holding off the time between obtaining the watched
folder's directory entries and commencement of file transfer to a few seconds
appears to render the possibility of even this condition to the point of
being very highly improbable (virtually impossible).

Wirt Atmar

* To join/leave the list, search archives, change list settings, *
* etc., please visit http://raven.utc.edu/archives/hp3000-l.html *