Permission to make digital/hard copy of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication and its date appear, and notice is given that copying is by permission of ACM, Inc. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. DOC 95-10/95 Savannah, Georgia, U.S.A 1995 ACM 0-89791-713-8/95/0010
An online documentation system can provide network operators with the information to conduct network management by exception. The AT&T PersonaLinkSM Services Network, based on innovative TelescriptTM technology, never before used in an operating network, had network node performance characteristics that were unknown at the outset. To insure the network could operate under normal conditions as well as failures, a great deal of effort went into the design of monitoring and control features for the network management system (NMS).
Personnel at the Network Operations Center (NOC) operate the network remotely through the NMS, using methods and procedures that were developed for this service. The online documentation system serves as the network operators' reference for carrying out those procedures.
The network supports messaging among new types of personal intelligent communicators. These
devices employ General Magic's MagicCap software to support the specialized Telescript
communication language. MagicCap enables a device to create Telescript agents that move around the
network and perform tasks as directed by their scripting. The details of this technology are beyond the
scope of this report (for general information, see [1]). The significance for network operations lies in
the need for a Telescript engine (TSE) at each location where Telescript agents must perform tasks.
Every MagicCap device, and every node in the network, includes one or more TSEs. The TSEs
support enriched communication (including sound, animation, and graphics) between MagicCap users
(see Figure 1).
One of the Network Operations manager's overriding concerns was whether he would be able to support 24x7 service (continuous operation 24 hours a day, seven days a week), using the tools provided by the network management system, with guidance from the methods and procedures documentation. One particular concern, the Telescript engines in every network node, was handled by an intensive development of operations support software to monitor and control the execution of the engines for day-to-day operation of the network. The operations support software, developed by AT&T, provides interfaces between all of the operating software in each node and the network management system.
The network management system, integrated from a suite of
standard network management software packages, runs in a
separate server at the network operations center (NOC).It
includes a trouble ticketing system that is used to track error
events and routine operator actions. The NOC connects to the
node, the customer support center, and the development labs via
leased data lines. Figure 2 presents a view of the overall
network management architecture.
The Network Operations personnel who operate, administer, and maintain the network need documentation to help them in their day-to-day support of network service. The role of documentation is to define procedures covering all aspects of running the network. To perform this job efficiently, an integrated online documentation system was considered indispensable. A requirements analysis and technology study was undertaken to clarify the documentation requirements and devise methods for satisfying them. This study was conducted quickly, so that its results could be used to plan the documentation for Release 1 (the first release of the service).
The problem at hand was to determine the best tools to develop and view the online documentation, and also provide printed documents. The areas to be investigated included documentation standards, development tools, and output presentation requirements and tools
The required features constituted the minimum set of capabilities needed to discriminate between competing documentation tools-that is, they were the disqualifiers. This list is not exhaustive; some necessary capabilities were not on the list because every product considered had them, so they served no useful purpose for choosing tools. Any product missing one of these capabilities was eliminated from further consideration.
The required features were divided into two categories:
• Document Authoring Software
This included the features for developing the documents and creating online and printed versions.
• Viewing Software
This included the features for accessing and displaying the online documentation, navigating through hyperspace, and creating notes and bookmarks, etc.
Table 1 lists the requirements by category.
Category | Requirement |
| Document Authoring Software | a. Can incorporate files dynamically. b. Imports FrameMaker, ASCII and troff files. c. Supports NMS platform and Windows PCs. d. Supports hypertext documents for online documentation. e. Can be expanded beyond initial configuration, by adding options or third-party add-on products. f. Document chunks can be incorporated into both online and print versions with no changes. g. Vendor has sufficient financial strength and market share to support the products. |
| Viewing Software | a. Can be called up independently of online help screens. b. Can be implemented on NMS platform or separately. c. Can be linked to context-sensitive help for network management applications. d. Files are read-only. e. Files can be annotated. f. Supports NMS platform. g. Supports hypertext documents. h. Supports indexed access. i. Supports paged tutorial mode access. |
The optional features served as points of comparison for tools that passed the first comparison, and also could be desirable for enhancing the quality of the final documentation products. Table 2 lists the optional features.
Category | Optional Feature |
| Document Authoring Software | a. Automatic hypertext link generation tool is available. b. Automatic indexing tool is available. c. Exports ASCII files. d. Exports SGML files. e. Hypertext links can be generated automatically from index entries. f. Imports SGML files. g. Supports multimedia documents. h. Supports multi-platforms (Windows, Macintosh, other UNIX). |
| Viewing Software | a. Annotation can be preserved across updates. b. Annotation can be private or shared. c. Annotation files can be retrieved for document feedback/updates. d. Document contents presentation can be customized. e. Supports online user feedback. f. Supports bookmarks. g. Supports full-text search and retrieval. h. Supports hyperlink path retrace. i. Supports hyperlink path history. j. Supports multimedia documents. k. User interface can be customized. |
The review of potential documentation tools was conducted informally, in two stages. In the first stage, product and industry reports from sources such as Datapro Research and PC Magazine were reviewed for basic information about product capabilities. The objective of this survey was to encompass as many products as possible, and insure that no viable alternative was overlooked. Products ranged from low-end desktop publishing to high-end typesetting systems. The products considered included:
| • Atex • Compugraphics • FrameMaker/FrameViewer • Interleaf • Microsoft Word | • Pagemaker • QuarkXpress • Ventura Publisher • WordPerfect • Xyvision |
In addition, the built-in online help facility of the network management system was included in the study.
Products that failed to support any one or more of the required features were eliminated from detailed comparisons. In most cases, the eliminations were based on obvious deficiencies of the capabilities of the product compared to the requirements.
The initial comparison indicated that both Interleaf and FrameMaker had the basic features needed to support the project. The detailed comparison, using all of the criteria listed in Tables 1 and 2 in light of the information available at the time, showed that both products appeared to be sufficient for our needs, so the decision was based on the comparative cost and effort to acquire, set up, and use them.
The cost to set up Interleaf, including the recommended onsite installation support and annual support, was significantly higher than FrameMaker for our planned development environment. Interleaf prices for the document authoring and production tools are relatively higher than Frame Technologies, and prices for the viewing system are relatively lower. Thus, there was a crossover point for more than about 75 online documentation users, beyond which Interleaf would be cheaper to use than FrameMaker. However, for each additional writer added to the project, that point would have increased by another 30 users. From this comparison, it appeared that Interleaf would be more expensive for our project than FrameMaker.
Since the initial setup and annual support appeared to cost significantly more for Interleaf, the critical question became: did Interleaf offer a significant advantage over FrameMaker, in terms of higher writers' productivity, greater usability for online users, or easier integration with the online applications? It did not appear to do so. It was also relevant that FrameMaker had been selected by developers for creating internal documentation for the project, such as architecture, requirements, and design specifications.
Lacking compelling reasons for selecting Interleaf, we chose FrameMaker and FrameViewer 4 for our online documentation. Based on our schedule and budget constraints for Release 1, FrameMaker/FrameViewer made sense as the initial choice for our documentation system.
We developed a simple, uniform, and responsive format for all online procedure documentation, to minimize the operator effort needed to manipulate the documentation. The text fonts were chosen to provide good visual clarity on the type of display used in the Network Operations Center, and page size and format were designed to stand out from a visually cluttered background.
We broke the material into content units that were well-suited for presentation on the typical screen display for an operator. This was an iterative process, starting from an initial screen design and experimenting with layouts until most procedures fit ontoa single "page." The design process considered presentation issues such as organization into pages that fit well onto a display, window layout, navigation aids, font type and size, and so on. We used the KISS principle (Keep It Simple, Stupid) as a guide. Our aim was to help the user avoid getting "lost in hyperspace"[2]-in other words, following a string of cross-links into the documentation until there seems to be no simple way to get back to the starting point. We divided procedures into two categories:
• More-skilled and experienced personnel performing those tasks need primarily "memory-joggers" to guide them through the overall sequence of actions. For a given task, this sequence may consist of a series of steps in an expert-level procedure.
• Less experienced personnel can use hypertext links from an expert-level procedure to detailed procedures that are designed for newly-trained staff. The detailed procedures may have additional links to glossary entries, figures, and other aids that explain how to perform each step.
The organization of procedures, supporting materials, and links, resembles the structure in Figure 3.
There are several types of document window; Figure 4 shows examples of most windows (reduced in
size).
• Expert-level procedures appear when the operator calls up the documentation from the trouble ticketing system. They contain overall procedures that do not go into all the details of every step.
• Detailed procedures can be called from an expert-level procedure when the operator wants to see the details of a step. These are details that the operator would normally not need to see after becoming familiar with the expert-level procedure.
The first two window types use the same format. Although we distinguished between the expert-level and detailed procedures to some degree, in practice we found that almost any expert-level procedure may be used as a detailed procedure from another point in the system. The lack of a distinguishing "look and feel" between the two types of procedures gave us the flexibility to use them interchangeably as needed.
• Popup windows contain definitions of terms, tables, lists, and other detailed information the operator may want to use while going through a procedure. The operator can keep them on the screen as long as needed. They may appear in either a detailed or an expert-level procedure.
• Alert boxes are special popups that contain brief information, such as glossary items, which fit into a small window.
• Table of Contents windows contain only links to other document windows. A Master Table of Contents contains links to individual Subsystem Tables of Contents, which in turn link to procedures and other information.
• Index windows are used in the index document, which contains the alphabetized list of links to procedures by title, and links to keywords within procedures.
• Glossary windows define common terms and parameter values.
All of these documents are connected by hypertext links that enable the operator to move from one document to another with one mouse click.
The actual size of most windows (on a 19-inch display) is 7.0 inches (17.8 cm) wide by 6.5 inches (16.5 cm) high. The only exception is the alert window, which is 4.5 inches (11.4 cm) wide by 3 inches (7.6 cm) high.
The bottom of each page has large buttons that the operator can use to navigate through the online documents. They are normally light blue when active, and are greyed-out when they are inactive. Not all buttons appear in every window type. The buttons (and their functions) are:
• Index - Display the overall index for online documents.
• Contents - Display the overall table of contents for the online documentation (this leads back to the subsystem tables of contents).
• Close - Close all online document windows, leaving FrameViewer active in the background.
• Go Back - Close this window and return to the previous window or table of contents.
• Previous Page - Display the page previous to this one.
• Next Page - Display the page following this one.
These buttons are greyed out when appropriate-for instance, "Next Page" is greyed out on the last page of a procedure, and both buttons are greyed out for a one-page procedure.
Hypertext links, as noted earlier, are blue underlined text (shown as black underlined text in Figure 4). If the operator clicks on a hypertext link, FrameViewer calls up the information to which the link refers.
Glossary and Index windows include alphanumeric index tabs along the left margin, which are links to the corresponding pages elsewhere in the document. The index is the largest document in the set, containing nearly fifty pages. The tabs enable the operator to page rapidly through the index, searching for a reference, without the need to call up a search window.
The documentation is organized to provide access (hypertext links) to procedures directly from the network management system, from tree-structured tables of contents, and from an overall index for the documentation. An operator who needs to call up a procedure has several choices of access methods:
• If the operator is using the trouble ticketing system to work on a trouble ticket or incident report, a customized menu will call up the procedure that corresponds directly to the alarm reason code in the trouble ticketing system window.
• An operator who is troubleshooting a problem that does not have a trouble ticket assigned yet, or is performing routine maintenance activities (such as a daily backup), may use the top level table of contents to select a subsystem table of contents, then choose a procedure from those provided for the subsystem.
• When the operator is uncertain which subsystem contains the correct procedure (many procedures span several subsystems), the index provides a means to search for the procedure by alarm text, reason code, or keyword.
One key feature of this documentation system is its provision of what we call context-dependent help. Context-dependent help provides information tied to the contents of the current window in a network management application. The method we used to implement this feature was to link a shell command capability of the trouble ticketing system to the remote procedure call (RPC) interface of the FrameViewer software.
When the operator chooses the Online Documentation item on the customized menu of a trouble ticket or incident report window, the trouble ticketing system issues a shell command with the contents of the trouble code field as an argument. This is processed by a UNIX shell script to yield the filename of the corresponding procedure, which is passed as an argument to FrameViewer. FrameViewer opens that document file and displays it.
Operator training for the first release was conducted over a period of several months. It included formal training sessions presented by the developers, on-the-job training by network operations specialists, and a computer-based training (CBT) module on the use of the online documentation system.
The computer based training module available addresses the basics of the online documentation system. It resides on a desktop computer located in the main room of the network operations center. Operators can access the training module by opening an application icon in a desktop folder. It introduces the basics of navigation, taking the user through the documentation structure, page layout, and navigation procedures.
Developing documentation for an innovative type of network presented issues in planning, scheduling, and managing the effort, because the documentation effort was dependent on parallel development of the network, the communication devices using the network, and the Telescript software enabling devices and network to communicate. The network development organization was distributed across multiple locations, and document development was done on multiple platforms. All of this came together seamlessly for the first release.
Before we could begin the documentation, we had to prepare the FrameMaker, FrameViewer, and other hardware and software tools needed to develop the online documentation and integrate it with the operating software environment of the Network Management Center.
Preparing the development and test environment required completion of a series of tasks, including:
• Obtain X terminals and software licenses for documentation writers and operators.
• Create a customized menu for the trouble ticketing system (TTS) for calling up online documentation, and demonstrate this to Network Operations (on the development NMS system).
• Create an online document directory structure on the development computer and network management computer, ready for population with online document files as they are delivered.
• Define FrameMaker templates for each of the document window types.
• Settle questions about how to use FrameMaker hypertext features to support easy navigation through the documentation.
• Define the file and directory system so that writers can use cross-references between procedures in separate subsystems.
• Solve cross-platform portability problems, such as discrepancies between implicit font names used by UNIX and Windows versions of FrameMaker.
One of the early objectives of procedure development was to estimate the total number of procedures in the documentation system, because that would directly determine the scope of the development effort. For this purpose, we prepared a complete list of online procedure documentation with assistance from developers and the network operations specialists of Network Operations. Next we organized the list of procedure documentation into Operations, Administration, and Maintenance categories, then had each development group and Network Operations review it. Based on this list, we made assignments for the writers and juggled the workload from time to time.
The technical content of procedures was "owned" by the developing organization, meaning that they were responsible for the completeness and accuracy of all information contained in the procedures under their technical ownership.
• Developers were the technical owners of all source material for procedures related to software they developed and released to Network Operations. This included software developed by General Magic, Inc. and incorporated into the final system by internal developers.
• Network Operations was the technical owner of all source material for procedures related to the network management system local adaptations, configuration, and similar material under their control.
The development process involved a series of steps by people in three organizations (Development, Network Operations, and Network Operations Documentation). The formal development process consisted of the following steps for each procedure:
1. The developer or technical owner delivered source material to the writer.
2. The writer prepared a draft of the procedure from the source material.
3. The writer delivered the draft to the technical owner for review of its technical completeness and accuracy.
4. The technical owner returned comments to the writer.
5. The writer prepared the revised draft for review by Network operations.
6. Network Operations reviewed the procedure for compliance with Network Operations policies and practices, and returned comments to the writer.
7. Network Operations conducted Operational Readiness Testing (ORT) of the entire network and the network management system, using the procedures as delivered.
8. The writer corrected any problems or errors found in ORT.
9. The finished procedure was delivered for production.
Progress of the development activity was measured by the number of procedures:
• with complete source information
• in development review
• development review complete
• in NOT review
• NOT review complete
• ready for ORT
• ready for service
This information was revised weekly, and circulated to the Network Operations manager and staff.
Draft procedures were delivered for testing by installing the procedure files on the NMS computer system.To complete the turnover of Release 1.0 documentation to Network Operations, we put all documents through quality assurance checks, placed them under source control, and used the entire set to generate up-to-date system level document files (such as the index).
Every online document file (including TOCs, expert-level, detail, and popups) was reviewed with the following checklist:
All comments from developers and Network Operations (NetOps) are incorporated.
Each file has been inspected by the writer for the following:
- no unavailable fonts
- no formatting glitches or typos
- no unresolved questions waiting for information from development or NetOps
- all hypertext links work properly
- index tags created for title, alarm reason codes, parameters and keywords in text, as applicable
- title in each procedure is consistent with table of contents (TOC) and index
- each alarm procedure title contains the alarm code
- glossary tags are created for keywords, alarms, and parameters
- symbols are turned off
- borders are turned off
- rulers are turned off
- spell check has been run
- file permissions are set properly for all files on network management system
- document display window fits the page size
- saved in view-only format
- latest template is applied
- each file has been placed under source code control
- all copies of files in the shared directories on development system and network management system are identical to the version under source code control
- master table of contents is up to date for all subsystems
- subsystem tables of contents are up to date for all procedures and files, and sorted alphabetically beneath each subheading
- final index is generated
- final glossary is generated
Final delivery was complete when all of the review steps were finished and the revised procedure files were installed on the NMS computer.
For the first four months of this effort, this author conducted the requirements analysis, selection of tools, and specification of the documentation architecture. At the end of that time, the scope of the effort was better known, and it clearly called for at least four full-time writers, with the author acting as project leader. By the fifth month, three additional writers were recruited for the project: Frank Rees, Elaine Artis, and John Godfrey (a fourth writer, Mike Dessoye, was recruited to document features scheduled for a later release). Within two weeks of hiring, they began to attend training and orientation seminars conducted by the software developers.
The documentation package included a tutorial document for new operators and other Network Operations people. We began to prepare for writing the documentation by combining the training for the writers and Network Operations staff with the research for this tutorial. A six-week series of seminars conducted by developers provided much of the detailed information we needed to write the tutorial sections. Understanding the system well enough to write tutorials would provide the foundation of technical knowledge the writers needed to create good procedures. By the time they finished the first draft of the tutorial document, the writers were immersed in procedure development.
Work on the second draft of the tutorial was postponed until after the network was in service, and was finally issued in April, 1995. Although the initial plan called for the tutorial to be available online, the tutorial was finally issued as a print document after it became clear that it was neither necessary online nor well-suited for online use.
The network naturally divided into a set of about twenty subsystems of varying size and complexity. We allocated the workload by subsystem, with one writer devoted full-time to the most complex subsystem, while other writers were responsible for up to five more or less closely-related subsystems. We occasionally levelled the load by trading assignments, in order to meet deadlines.
Following the initial release of the network service, we reorganized the documentation team to support Network Operations documentation for ongoing operations and future development. Our agenda was:
1. Complete any outstanding revisions (including changes resulting from initial experience with live operations) to the Release 1.0 documentation within six weeks of first service.
2. Set up ongoing methods and procedures analysis at the network operations center to capture changes to operating procedures and revise the documentation accordingly.
3. Update and revise the tutorial document to meet the needs of new operators learning the system.
4. Support near-term point releases to the network.
5. Plan support for future releases of the network, and for the documentation of other products and services that Network Operations may be called upon to support in the future.
1. Presenting Magic Cap: A Guide to General Magic's Revolutionary Communicator Software. Knaster, Barbara, Reading, MA. Addison-Wesley Pub. Co. 1994.
2. Scheiderman, Ben "Reflections on Authoring, Editing, and Managing Hypertext," The Society of Text, Edward Barrett, Ed. Cambridge, MA. MIT Press, 1989