X-no-Archive:yes
It is a problem with NT; the rename fails. I did not say that this is a
problem with UNIX, nor could I, having no experience doing this going TO a
UNIX system.

Now, Wirt mentions the trigger file. I inherited a script that uses this
approach, but to tell the script that files are available for download,
which I gather could be said to make the process asynchronous in some sense.
My question is, what to do with the trigger file? Delete it? If so, when?
Immediately? What if my batch job then blows up for some stupid reason,
presumably a design flaw or failure to plan for problems or foresee the
unexpected? Can I just rerun the batch? If I still have the trigger, no
problems. If I deleted it immediately after detection, then I need a good
way to recreate the trigger file, which fortunately can be trivially easy on
the 3000, among others. But those who baby sit production do not always have
even the authority to build or touch a file in the production account.

In the script I inherited, the script waited until the end to perform any
clean up, including deleting the trigger file. One night, it died, because
it could not connect to the SMTP server to send out its status email. So,
someone had to sign on, and troubleshoot, confirming that everything worked,
but no one thought about the trigger file. So the next night, as soon as the
script started, it saw the trigger file, got whatever files were there on
the foreign host, and ran to completion. The owners of the application on
the foreign host were not amused, and demanded a fair bit of research to
prove that what was processed was what should have been. How did we know
that we did not get the files from the previous night? Not knowing the
application or batch processing on the foreign host, this was not an easy
question to answer.

And, does this trigger file only act as a dumb trigger, containing no
meaningful data? Plenty of mainframers want those header and trailer
records, even when the file is empty. Do I need to read those? Do I worry
about the timestamp on the file? If so, what happens with scripts than can
span a day? At that point, the scripts have to know more about when they are
ran, and that's not always a trivially easy thing to script. It's also
potentially fragile, if one hard codes times instead of scripting for the
job to figure out its own date and time when ran. Rescheduling the job can
break it.

So, I favor the rename approach on systems that do not wait until I have the
file to "catalog" it or otherwise make it visible to other processes. I have
the data file when I have the data file. Occam's razor. Also, looking for
the file itself is a pretty reliable way to do things on systems that DO
"catalog" the file, only after ftp finishes, such as our 3000s (so long as
failed sends do NOT "catalog" the file). That may be a thin argument, since
I do have to do * something * different when scripting for UNIX or NT, so
it's just a question of what I do differently.

I also assume the worst of human nature when I do these things, and have yet
to be disappointed. Nobody bothers agreeing on naming conventions upfront.
For instance, we needed to send a file created on our mainframe to a
financial institution, but encrypting it first. Since we do both mainframe
and Linux-based work for this customer, and have pgp encrypting software on
our Linux box, the mainframe sends the file so as to be seen by a perl
script, croned on the Linux box. Sure enough, the name on the mainframe bore
absolutely no resemblance to the name that the financial institution wanted.
So, I provided JCL to the mainframer that would send the file with the last
node of the dataset name as the file name, then rename it to the basename
that the financial institution expects. And in some poor sense, this
provides the mainframers with the name that the financial institution uses
as well. I then merely detect the file, encrypt it, which gives it the pgp
extension, and send. Everyone else has what they want. In the rarer case
when both sides actually agree on a name, that name has contained an
extension, so it was trivially easy, although not intuitively obvious, to
send without the extension, then rename to include the extension after the
send.

Greg Stigers
http://www.cgiusa.com

-----Original Message-----
From: Ken Hirsch [mailto:[log in to unmask]]
Sent: Thursday, October 04, 2001 1:54 PM
To: Stigers, Greg [And]; [log in to unmask]
Subject: Re: hp9000 -> hp3000 -> hp9000

Is this really a problem with NT?  If so, what are the symptons?  Does the
rename fail or does it work, but the file not have all the records?  With
Unix, it doesn't matter if the buffers are on disk or not (unless the system
crashes).  The file system treats the records in the buffer as part of the
file anyway.  The records are all there, logically, as soon as FTP closes
the file.

I'd be surprised if this was really a problem with NT, except perhaps very
early versions of NT.

* To join/leave the list, search archives, change list settings, *
* etc., please visit http://raven.utc.edu/archives/hp3000-l.html *