(*)

Y2K Technical Firefighting Symptoms/Causes/Fixes

(*)

(*)

Copyright ©1999, 2000 by Evan V. Robatino. All Rights Reserved.

This document may be freely copied provided that it is reproduced in its entirety, including the copyright notice.

Note: The Y2K rollover has now taken place, and the initial effects were much milder than I anticipated. Notice that I said initial effects - there have already been a number of "small" Y2K-related computer incidents since the year 2000 began. Fortunately, the power and telecommunications (including the Internet) infrastructures have remained intact, making recovery from any future incidents much easier. Fix On Failure (FOF) is possible when a Y2K incident is "cosmetic" in nature (e.g. an incorrectly displayed year in a Web page) and may be possible when the resulting damage from such an incident is minor.

The output of any program runs in the near future, paticularly of programs that process dates, should be closely checked for obvious anomalies that may be Y2K-related.

Persons responsible for database processing that involves dates should be extremely cautious and closely check the database contents and the results of any batch runs for possible corrupted date fields until full confidence in post-1999 results can be established.

This is a "techie" document directed at the people who (like myself ;-) ) were working long hours on or about 1/1/2000 or other Y2K-critical dates watching for (if not actually having to put out) these long-dreaded Y2K fires. I myself worked two shifts over Y2K rollover weekend baby-sitting a particular file transfer product to ensure it still worked correctly (it did).

Introductory documentation: Jocelyn Amon's excellent Year 2000 Articles, particularly The Y2K booby-trap and other related myths and New Year 2000 Bugs Found in Internet Web Pages. (As an aside, I hope there aren't too many mission critical JavaScript- or Java-based applications with Y2K errors or we're in even more trouble than I anticipated! These types of bugs were quite abundant immediately following rollover, but fortunately were mostly cosmetic.)

Also see:

Introductory note: This document is primarily directed at organizations or persons who have "completed" their Y2K remediation efforts, only to find that they missed something on or after one of the Y2K "critical dates" (list supplied by the MITRE Corporation) and then have to scramble to fix it (or are finding bugs during remediation but are having trouble figuring out causes/fixes). Those people have my sympathy, since they're at least trying. Anyone who is reading this document before Y2K arrives and is foolish enough to expect to be bailed out by it come 1/1/2000 (to forego making preparations for Y2K with the idea that you can just "Fix On Failure" (FOF) when problems start popping up): Forget it. Please do yourself a favor and begin Y2K remediation immediately, so that you'll have a fighting chance to rid yourself of the sneakier time bombs that you couldn't find and defuse in time. Remember that it's not just the software you'll have to debug and repair and test (testing has been found to take a minimum of 60% of most Y2K remediation efforts) - it's also the damage done to data files that would not have occurred had you fixed and tested the code to prevent the "bomb" from going off. Unless you have only one or two PCs, no non-Y2K-compliant application software, and are personally technically competent enough (and have the time) to fix your own systems (at a time when help desks worldwide will undoubtedly be deluged with trouble calls, you will be on your own), FOF at the last minute is not an option. Enough said.

For the wise who did prepare for Y2K but of course didn't find every last hidden glitch (welcome to the club; I'll probably find myself in the same position despite my fanaticism on the subject of Y2K prevention ;-) ): Start learning how to use the symptoms from the many mysterious outages and problems that will arise to quickly home in on the causes of these troubles. Prepare for the inevitable FOF scenarios stemming from the bugs that will come crawling out of the cracks on or about 1/1/2000 despite your best extermination efforts because they will happen, like it or not.

I'm attempting to do here what I haven't seen anyone else on the Web do yet - list symptoms of Y2K-related hardware and software malfunctions and point to possible causes and cures. Of course, it's plainly impossible to write a totally comprehensive list of symptoms and cures for a problem as widespread as Y2K (such a list would quickly grow to something like millions of entries), and which hasn't even happened - yet. All I can do is list some anticipated symptoms and suggest avenues of approach that will start you thinking of how to look at similar problems you may (or should I say will?) encounter. Any troubleshooter in any profession - systems analyst, physician, auto mechanic, et al - works by gathering symptoms and then correlating them against his/her own knowledge and experience to come up with remedies (or at least possibilities for workable remedies).

Exactly how high will the "alligator index" (as in 'When you're up to your butt in them, it's long past time to drain the swamp') get? Nobody knows for sure, but I'll take a Simple Wild-Assed Guess (SWAG) and say "Pretty darned high". The best I can do here is to attempt to cover some of the most likely observable symptoms of Y2K problems and suggest avenues of investigation that will, hopefully, lead to fixes (assuming, of course, that electricity is still available to power the systems on which the fixes are to be made and that we still have at least our basic food, water, shelter, etc. needs being met so that we can work these problems relatively undistracted). In my own experience in tracking down software problems, I quickly learned that it was totally pointless to memorize problems and their corresponding solutions; there are simply too many problems and solutions for this. What did work, after gaining basic competence in the hardware/software platform being worked on, was learning how to dig for information:

Examples are:

You can probably think of other examples.

In the table at the end of this page, I've attempted to list a number of different Y2K-caused symptoms and possible causes and cures. These are based on:

  1. Actual Y2K "time machine" testing (setting the system clock ahead to the year 2000 to see how the code reacts, or using a system "time warp" tool to simulate operation in the year 2000 to cause this effect), or
  2. Bugs found in existing code that would have resulted in Y2K problems if they hadn't been found and fixed in time, or
  3. What could reasonably be expected to happen to unremediated code.

Again, they are not (and never can be) completely exhaustive and I haven't had the time to think of more examples - yet. (Wait until around 1/1/2000 and I'll probably have a few more ;-) .) My basic intention is to put in "teasers" to get you thinking:

Keep in mind, also, that Y2K problems can be caused by year 2000 data that's fed to a system before the year 2000 is actually reached (examples are reservation systems and financial systems beginning Fiscal Year 2000 processing in 1999), as well as by the system clock being set forward to (or rolling over to) 2000. Such incidents have already occurred and will probably increase (or be reported) more often as 1/1/2000 is approached.

Once we're past the Y2K rollover point, don't assume that all is well because a particular program or system hasn't broken due to Y2K just yet. A reset, reboot, or install that happens for the first time in 2000, even if it's weeks or months after rollover, could be the trigger for a Y2K failure. A perfect example is a Windows PC with a non-Y2K-compliant BIOS. If left powered on and the operating system keeps running past midnight 1/1/2000, the Windows clock will roll over just fine and keep going, theoretically forever - until the system is rebooted. At that time the BIOS year value of 1900 is first seen by the operating system, causing Windows to substitute a year value of 1980, 1984, or whatever. Moral of the story: Changes, restarts, etc. will have to be made with extreme caution after Y2K rollover until each change or restart type has been done at least once without ill effects.

Legal disclaimer: This is a good faith effort on my part to help technical people who will shortly be wrestling with live Y2K problems. However, I cannot and will not be responsible for consequences resulting from end users taking actions resulting from the advice on this Web page. When possible, you should consult the vendor of any software suspected of causing Y2K problems. (Definitely contact the vendor(s) of any software you use that has a date-dependent license and for which you plan to do any "time machine" testing so that you can obtain a temporary testing license for that future period; your current license may well not be valid for the year 2000 and later.) I attempt to keep the advice on this page accurate and up-to-date so as to be of the best practical use to the people on the front lines, but of course accuracy cannot be guaranteed. Constructive suggestions for its improvement are always welcome.

One last word: Keep in touch with your hardware/software vendors; check with them from time to time to see if any new Y2K fixes applicable to your environment have been released. It's an unfortunate fact that Y2K compliance (no matter how you define "compliance") is a moving target (e.g. witness Microsoft's constant issuing of new Windows NT 4.0 service packs (they were working on SP6 last time I looked) and Windows 95 fixes - not to harass Microsoft, since they're struggling with this problem as much as anyone, but to show just how tough it is to nail down Y2K compliance). Vendors keep finding new Y2K problems they have to issue fixes for. Keep your information up to date.

Acknowledgements: Where possible, I have placed the names of people who discovered problem fixes next to the fix descriptions themselves. Many contributors, of course, are implicitly recognized as the authors of the various Web links I've taken the liberty of referencing. If I have failed to recognize anyone whose contributed to the fix list below, please let me know so I can add your name as appropriate. Much of the information below was obtained by regular reading of the comp.software.year-2000 Usenet newsgroup (you have to do some separating of signal from noise, but some useful information does come through from time to time ;-) ). Another virtually indispensible source of technical Y2K information has been Peter DeJager's Y2K mailing list, which I gladly pay $50(American)/year out of my own pocket to subscribe to. (If you're interested in a trial subscription, send an e-mail message to amy@year2000.com with the word "SUBSCRIBE" in the Subject: line; there's a 30-day trial period before you have to begin paying.)

Enough lecturing. Here's the table, broken down by hardware/software platform, with a local link to each platform:

Generic (may occur on several different platforms) Network and Internet
Embedded Systems MVS (IBM's Multiple Virtual Storage System)
UNIX / C / Perl Java
JavaScript IBM compatible PCs (includes Windows operating systems)

Y2K Symptoms and Causes/Fixes

Platform Symptom Cause/Fix
Generic (may occur on several different platforms) User signing onto an online system on or after 1/1/2000 and being told "Password expired" (or "Userid revoked" or similar message) - generally occurring where system requires periodic password changes. Security system is not Y2K compliant - thinks user's ID is 99 years old and revoked it. Contact security administrator to reinstate ID.
System displays year "19100" (perhaps truncated to "1910") or has this value in a variable. (Java and UNIX platforms.) Program or script has concatenated "19" to the left of the value returned from the getYear() method in Java or tm_year variable in UNIX or "$year" variable returned by the localtime() or gmtime() function in Perl. Fix: Add 1900 (don't concatenate "19") to original method/variable to get correct four-digit year.
Fiscal-year-based accounting system/program crashes or yields invalid results (e.g. puts transactions in wrong fiscal year) between late 1998 and early 2001. Program is exhibiting the "Jo Anne Effect" (see The Jo Anne Effect or Cory Hamasaki's DC Y2K Weather Report Number 91 for a detailed description). Probable cause is an erroneous sort or logic check on two-digit years in data for fiscal years starting in 1999 and ending in 2000.
Network and Internet E-mail bounced back to sender during 12/31/1999-1/1/2000 period. Mail server in time zone in year 2000 thinks mail from 1999 is 99 years old (and therefore invalid) and bounces it back. Problem should clear in 24 hours when all mail servers are in the year 2000 - contact mail server administrator if it persists.
Your local TCP/IP network crashed on or about midnight 1/1/2000. May be an SNMP (Simple Network Management Protocol) problem caused by non-Y2K-compliant "intelligent" bridge, router, brouter or gateway hardware or software. Some additional things to look at:
  • The RTC (Real Time Clock) or BIOS in one or more network processors may not be Y2K compliant and be feeding incorrect date information over the network.

  • A time synchronization program may have a Y2K bug.
(The TCP/IP protocol itself, in general, is Y2K compliant but components implementing TCP/IP may not be.)
Embedded Systems Peripherals controlled by system (e.g. power distribution systems, factory robots, elevators, PBX systems, UPS, etc.) begin malfunctioning at or near midnight 1/1/2000. System has a date dependency and is malfunctioning on rollover from the year "99" to the year "00". Fix: Reprogramming and/or replacement of system by qualified engineer. For more background information, see Mark Frautschi's Embedded Systems and the Year 2000 Problem.
MVS (IBM's Multiple Virtual Storage System) Program crashes at or shortly after midnight on 1/1/2000; dump shows ABEND (Abnormal End of program) stemming from execution of TIME macro (SVC (Supervisor Call) 11 instruction), e.g. TIME DEC,LINKAGE=SVC format. Program may be executing old form of SVC 11 call and getting back a year of "00" (or "100") and not handling it correctly. Fix: Program should be adding 1900 to the returned year value to get a correct four-digit year. A correct example for an assembler program is:
    *  TIME MACRO RETURNS DATE
    *     IN GENERAL REGISTER 1.
             TIME  DEC,LINKAGE=SVC
    *  DATE IN 0CYYDDDF PACKED
    *     DECMIAL FORMAT. 
    *  C=0 IN 1900s, C=1 in 2000s.
    *  YY = LAST TWO DIGITS OF YEAR.
    *  DDD = DAY OF YEAR.
    *  F = SIGN FOR PACKED DEC. FORMAT.
             ST    R1,ORIGDATE 
    *  ADD 1900 TO DATE (FOLLOWING AP
    *  INSTRUCTION LEAVES DAY OF YEAR
    *  INTACT).
             AP    ORIGDATE,=P'1900000'
             ST    R0,TIME
             .
             .
             . 
             ORIGDATE DC    F'0'
             TIME     DC    F'0'
             
PL/I load module returns wildly invalid values from DATE function calls on or after 1/1/2000 (e.g. month=14, day=43 (sic)). Old PL/I load module using non-Y2K-compliant runtime library modules (documented problem with DATE() function, library module IBMBJDTA - that pesky SVC 11 problem again). It may be possible to locate the offending module(s) by scanning for string:

X'921DD05A9200D0554F10D050'

If you find this in a module then it probably has the "bad" DATE module and should be relinked, particularly if that string is in IBMBJDTA or the module has not been linked for years. It can't be guaranteed that this will find all versions of the bad module, but it matches the ones that have been found.
Fix: Apply PTF UN56802 for OS PL/I Ver. 2.3, or the earlier applicable PTF for OS PL/I Ver. 1.x to update the IBMBJDTA library routine in SYS1.PLIBASE. Then relink all affected programs (it should not be necessary to recompile); relinking mostly consists of using "REPLACE IBMBJDT1" to pull in the correct library routines. For a complete description of what needs to be done, see the IBM Year 2000 Alert Regarding Use of PL/I Built-in Function Date( ) -- Actions Required article. (Grateful acknowledgements to Andy Wood of Westpac Banking Corporation, Rob Watson of IBM, and Cory Hamasaki for picking up the details of what may have been the root cause of the Meltdown: Australia problem and particularly to Rob and Andy for providing some much-needed clarification of the correct fix(es).)

Load module ABENDs with a storage violation after 1/1/2000 is reached or when fed data dated in the year 2000. Check for out-of-range index or subscript generated by date-related code that may be causing storage fetching/overwriting from/to an invalid address.
UNIX / C / Perl Script/program error - year of "19100" (possibly truncated to "1910") assigned to script/program variable. See "19100 fix" above for fix.
cron entry is not deleting log files whose names (in SYYYYMMDD.nnn format, where "S" is a literal "S", "nnn" is a sequence number and YYYYMMDD is the four-digit year, month, and day) contain the year 2000. cron entry contains 'S199[0-9][01][0-9][0-3][0-9].[0-9][0-9][0-9]' to match the file name and thus doesn't match the year 2000 embedded in the file name, so post-1999-dated files aren't erased. Fix: Replace hardcoded "199" with "[1-9][0-9][0-9]" to match all possible years between now and 9999.
Java Program error - year of "19100" in program variable. See "19100 fix" above for fix.
JavaScript Script error - year of "19100" returned in JavaScript variable (probably occurring in a Netscape Navigator browser). Script is concatenating "19" to left of value returned from call to getYear() method. Fix: See JavaScript and the Year 2000 (Y2K) Problem and Year 2000? Not a Problem! A Cross-Browser Solution for Dates (JavaScript).
Script error - year of "192000" returned in JavaScript variable (probably occurring in an MS Explorer browser). Same as above (getYear() method returns four-digit years starting in 2000 for Microsoft Explorer browsers, so symptom is slightly different). Fix: See above.
Script error - year of "3900" returned in JavaScript variable (probably occurring in an Internet Explorer browser). Script is adding 1900 to value returned from getYear() method - the developer almost did it right, but Explorer increased the returned value to a four-digit year starting with the year 2000. Fix: See above.
IBM compatible PCs (includes Windows operating systems) On first boot-up in the year 2000, system date reverts to Jan. 4, 1984 (or 1900 or other post-1980 date before 2000). Cause: BIOS of PC is not Y2K compliant (has not set the century byte to indicate post-1999 dates) and has rolled back to the year 1900; operating system has probably then reset system date to first "possible" date. Fix: Use the MS-DOS DATE command (or the DATE/TIME icon in the Windows Control Panel) to reset the date to the current correct date (this should set the century byte), then reboot the system to ensure that the century byte remains correctly set. See Patrick O'Beirne's PC FAQ for more information.
Upon boot-up in the year 2000, system date has "jumped ahead" several weeks or months since the last time the system was up. Probably occurring on a 286, 386, 486 or early-model Pentium PC. System is experiencing "time dilation" phenomena. This is probably caused by an unbuffered Real Time Clock (RTC). See the TD Website for more information. Fix: Reset system date/time to correct value. Obtain patch (tdfix.exe program) from contact(s) listed at the TD Website.
COM2 port on PC has become inactive/inoperable. See "time dilation" above - probably caused by overwriting of COM2 status information due to buffering errors. It is unclear if resetting the system clock to the correct date/time will also fix this problem.
Year fields in Excel spreadsheet are not being calculated or displayed correctly (dates displaying as "#####" or as incorrect two-digit values). See Patrick O'Beirne's Spreadsheet FAQ for corrective action.
The year in the dates displayed by File Manager in Windows 3.1 or Windows 95 is displayed incorrectly for files created or updated in the year 2000. Bug in the original version of File Manager. Fix: Go to the Microsoft Year 2000 Resource Center page, click on the "Product Guide" link, then select your platform and download the updated Y2K compliant File Manager version and other fixes you may need for your platform.
Netscape browser displays Certificate Authority Is Expired error message when attempting to enter secure mode in the year 2000. Expired Certificate Authority (CA) certificate in the browser's software (many CAs expire at the end of 1999). Fix: This occurs in Netscape Version 4.05 or earlier browsers. Either upgrade the browser (easiest fix) or download an updated CA from the appropriate vendor. (This security problem also occurs with Internet Explorer 3.x or earlier versions but no error message is displayed. The fix is the same - upgrade the browser or the CA.) See Entrust.net's Root CA Expiration and Year 2000 or The University of California, Berkeley's Some Web Browsers Will Display Security Certificate Error Messages After December 31, 1999 for more information.

Forward any constructive criticisms, suggestions, etc. to erobatino@worldnet.spam.net. (Change the "spam" in the address to "att" before sending me e-mail. Flames are automatically redirected to /dev/null ;-) .)

(*) Back to Evan's home page.

Number of times this page was accessed since July 14, 1998:


Y2K Technical Firefighting Symptoms/Causes/Fixes / Evan Robatino / erobatino@worldnet.spam.net
(Change the "spam" to "att" before sending me e-mail.)