Hello once again all TurboIMAGE users:

This is an alert on a serious problem with TurboIMAGE Large File DataSets
(LFDS).

The BOTTOM LINE:
The current version of TurboIMAGE LFDS should NOT be used:
It is seriously broke.


This update is late by at least 3-4 weeks.  List of excuses is long;
primary being that our team is at the peak of a flat-out race to complete
our TurboIMAGE-2-SQL/Server migration within the next 3 months;  before we
fall even further behind schedule (you don't want to know (heck: *I* don't
even want to know)).  Be that as it may, I should have made the time to do
this a few weeks ago; when I was first made aware of the last piece of
important info;  i.e.:

While strictly speaking you cannot run into this problem by accident during
normal operations, we also didn't think anybody would use UNLOAD -
schema_change - RELOAD to go to TurboIMAGE LFDS.  Bad assumption, since I
recently got first word that at least a few sites HAD gone that route.  For
whatever lingering responsibility I have fallen down on as temporary /
left-over / place-holder SIGImage/SQL Chair, I regret not getting to this
until now.  Hopefully better late than never.

And in fact there is NO WAY I have time to do tonight what I have been
hoping to get time to do:  Pull all the technical details together in a
coherent and comprehensive post.  Since I am about to go out of town for 4
days highlights will have to do.  Others in the technical community are
aware; and of course if you have further questions and are one of those
sites who still have an HP software support contract, you could try a
radical approach:  You could ask HP for an update, and post whatever
factoids you glean (if any) to 3000-L and/or the OpenMPE list (I will post
this to both).


ANYWAY:
Here is an ABBREVIATED summary:

(1)  Way back some time in 2004 I first heard about some problems with LFDS.
At the time I got the impression that they were just performance issues.  I
posted said impression on both (I think) 3000-L and the OpenMPE list.  I was
right about the performance issues, but WRONG about that being all that was
amiss.

(2)  The key problem:  If an IMAGE entry spans the 4 GByte point in a LFDS,
a DBGET of that "spanning" entry will return some correct data, plus a
portion or all of the UserLabel, and possibly additional data.  Similarly,
DBPUT of a spanning entry will store some correct data, and then write user
data into some or all of the UserLabel and possibly into another entry.
IOW, any entry than spans a multiple of 4GB will be corrupted by DBGET, and
will corrupt data during DBPUT.  Due to the UserLabel, a LFDS will always
have one IMAGE block that spans the 4 GB threshold; UNLESS the dataset
BlockLength results in an MPE record size of 128 half-words (a rare event
these days, I would think).  The problem will also occur again at the EIGHT
GB threshold, for ALL record sizes.  See again above bottom line:  LFDS is
seriously broke.

SIDEBAR:  There was another bug in LFDS that caused a system abort that has
already been fixed in patch MPEMXQ1.

(3)  I don't claim to know the low-level technical details, but I understand
fixing this problem is non-trivial, and involves fairly pervasive
modification of the underlying IMAGE code;  i.e.:  Fixing the current
implementation of LFDS is a BIG job.  Initial word I got a few months ago
(before I knew somebody had actually stumbled on the problem) was that HP
was hoping to have a patch out by the end of 2004.  Recent impression I got
is that not only did it not make the end of 2004, HP is now having to
regroup at least to some extent on the planned fix; and patch delivery date
is now unknown.  Given HP is still AFAIK scheduled to turn out the lights on
the HP e3000 at the end of 2006, we are getting very late in the process to
get a major patch still (probably) some months down the road; that will
likely be difficult to get comprehensively beta tested.

(4)  Looking ahead to end-of-HP-support, IMO the over-riding objective is to
have HP leave IMAGE as stable as possible, and with all recent fixes
thoroughly tested.  In retrospect it seems that it would have been better to
complete the original design idea on Jumbo Datasets, allow Jumbos to support
DDX, and permanently forget about using LFDS in TurboIMAGE.  Whether in the
current circumstance it is too late to revisit that idea, I am not qualified
to say.


SIDEBAR:  Our site has absolutely no dog in this hunt:  We are still happily
running on TurboIMAGE C.09.02, and will do so until we switch over to
SQL/Server.  We will not come close to the 4 GB dataset limit before we make
the jump.

That's all I have time 4 b4 escaping tonight.  If you have questions about
this problem, most likely I will not be able to take time 4 anything but a
cursory reply (and I probably don't know the answers anyway).  But please
post questions if any to 3000-L and/or the OpenMPE list.  And contact HP, if
you still have a contract..

FOOTNOTE:  It has been over a year since my last expose....  Guess it was
time...

Ken Sletten