Paper! The Problem is Paper

As I've at least hinted at over time I work as a computer programmer. Where I'm at now we have a large number of programs that run every night (m - f) to update the system.  Those programs, literally hundreds of them, run on different platforms - a mainframe and a server - and run in a defined order. 

We have software that defines the order and controls the process to insure the defined running order is maintained. Besides defining the order and controlling the flow of the programs the software also automatically contacts programmers if a program fails.  Addditionally, we have a 28 page long document that graphically lays out the order.  When a change is made in the controlling software we always, always update the document.

Last Friday one of the programs was put on hold.  It was determined that there were only 3 programs that ran after that program. The person that made that determination used the 27 page long document.  Yes, the 27 page document.  That's not typo. The programmer was using a copy of the document that was printed prior to October 1st when the 28th page was added. 

The new programs on page 28 depended on the completion of the program put on hold to run. Therefore those programs didn't run Friday. Because they didn't run, they didn't fail and there was no automated contact. I only discovered it because we were processing some new data on Friday night and I checked on the programs out of curiosity.  I spent several hours yesterday determining what happened and resolving it.

The programmer that put the job on hold - at the direction of managers - should not be using a printed copy of the document. The updated document is readily available on our network and should have been checked.