A recordkeeping critique of document management systems
Barbara Reed, 1995

Unpublished

Introduction
Document management software provides tools of increasing sophistication. In response to marketplace demand developers are adding more user definable fields with which to surround the document content. With such additional functionality document management systems are moving beyond being concerned with facilitation of document creation and evolving into packages which aspire to manage evidence of business activity - into records management systems. They appear to be doing so by default, by following the demands of the users of such systems. Little overt attention is paid to the emerging consensus of the recordkeeping professions on what constitutes evidence of business activity over time. An analysis of the perceived aims of document management systems from a recordkeeping perspective highlights some areas which require enhancement in most document management packages. Only with such enhancements and attention to emerging recordkeeping specifications will document management packages be able to fulfil the promise of serving organisations’ needs for reliable, accessible and useable electronic records.

The attraction of document management systems is clear from the market penetration they have achieved. They hold out to organisations the prospect of regaining control over that portion of the organisational knowledge base which is held in ‘unstructured’ documents. Formal control of the processes which capture documents as records has diminished rapidly with the distribution of powerful desktop computing to individual end users. Empowered by these facilitative tools, users have grasped the potential of the technologies and bypassed the old systems which served as mechanisms to ensure records of activity were routinely captured. As a result, the documentary base of the organisation is fractured and fragmented, residing in individual desktop computers with no consistent or coherent identification and minimal accessibility.

Document management packages address these issues, through their multi-faceted functionality which promises enhanced co-operation between individuals formed into workgroups, shared storage space creating a corporate store and powerful retrieval tools to enhance accessibility. They promise the capacity to alter the ways individuals cooperate and share information in the document creation process. They offer prospects of automating and imposing control on that part of the workplace involved in non-routine transactions, where complex rules applied to ad hoc situations have previously defied efficient automation.

They also implicitly allow organisations to assume that the resulting documents captured in such a system are capable of providing reliable evidence of the business activities and transactions which create them - they imply that they manage records. Establishing a documentary base which is reliable, trustworthy and which ensures that evidence of action is available for as long as it is needed, is a prerequisite to cost-effective, reliable and innovative reuse. All organisations are keen to maximise the use of the information resources already in their possession. The capacity to re-use chunks of information, to combine information in different ways and to fully exploit the data lying inert in documents holds great appeal. Are document management systems delivering what they promise?

As document management systems evolve from being facilitative tools to enhance document creation and move towards managing evidence of business activity, they are moving into the realms of records management systems. Records are information, data or documents which carry with them sufficient information about the context of their creation and their structural properties, and are embedded within the accountability framework of an organisation so as to ensure their integrity and authenticity for the period for which they are designated as accountable documents. The documents stored and managed in document management systems may be records. At present, most document management systems do not manage documents as records.

Similarly, electronic records management is generally not being achieved. Various innovative implementations combining software such as imaging, workflow, document management and traditional records management packages are approaching the functionality required for electronic records management. Few organisations can sustain the burden of supporting so many pieces of expensive software requiring clever implementation to ensure reliable, useable and accessible records. Establishing an organisation-wide recordkeeping regime involves determining how the business defines its functions and activities, business-specific rules about what records are required as evidence of action, and what external compliance and internal accountabilities will apply. These recordkeeping activities are analytic and cannot be performed by software. However, the resulting determinations should be able to be incorporated in the rules bases governing the operation of software packages. How well do document management packages address these requirements?

Document management systems available on the Australian marketplace can be analysed as offering four distinct functions:

• collaboration between members of work groups in the production of documents;
• capture of communicated documents as ‘locked’ or ‘fixed’ documents embedded or surrounded with sufficient contextual information which aim to make them reliable evidence of transactions;
• continuing management of those documents; and
• exploitation of enhanced information retrieval capabilities such as contextual searching, free text retrieval and weighted search algorithms.

Facilitate collaboration between members of work groups in the production of documents

Many document management packages focus on enabling individuals and the workgroup to exchange and collaboratively develop documents. Workgroups can be defined by the organisation in ways which transcend the organisational boundaries. Workgroups provide a level of modularity which is flexible, can be constituted at many levels and can be altered to suit particular projects or activities. Workgroups do not have to physically exist, but can be ‘virtual’ constructs which are enabled to undertake work through the innovative application of technology.

Document management becomes one of the enabling technologies available to organisations seeking to subvert formal and rigid structures. The facilitative nature of this role of document management seeks to enable the sharing of documents and the collaborative, cumulative pooling of expertise in undertaking complex tasks. Particular features in document management systems support this role. They include:

• extensive version control to ensure that the most recent document version is used to take development of documents forward;
• the capacity to review and annotate documents as they are being developed; and
• identification of which member of the group added, deleted, updated or approved amendments.

Document management systems also incorporate the capacity to specify the form of document. Form carries with it implicit meaning. We know at a glance, without reading, what a gas bill communicates - our eyes immediately go to the highlighted box located in a predictable place within the document. A contract too carries with it conventions which enable the form itself to be ‘read’ regardless of the content of the particular contract. Business letters, memos, circulars all have such intrinsic meaning in their form. Organisations can define many such documentary forms as norms within their own environment. The ultimate acceptability of forms of documentary communication is validated by the wider society - through courts of law and through validation of the systems which participated and surrounded their creation. At present, in our long transition between a paper based society to an electronic society, forms of communication are in flux and we are all participating in the development and mutation of currently accepted forms into new ones suited to the digital era. The study of documentary form has a long history contained within the science of diplomatics, a discipline which developed in C17, another time of transition between an oral culture to a literate culture.

Document management packages manage form through specification of document types, through enabling pro-formas and templates in word processing, email or by linkages to other software. Because form carries meaning, we need to track the types of form used within organisations in order to authenticate documents produced in the course of doing business. While specification of document types is incorporated into document management packages, this provides only a simplistic method, reflecting the latest, most current form. A knowledge base of the form of documents approved for use at a particular time is needed to ensure the contextual meaning carried through form is available as evidence over time.

When to capture a ‘fixed’ document?

Document management software packages are increasingly presented to organisations embodying more than just the capacity to facilitate the creation of documents. All incorporate the capacity to store, retrieve and manage the products of the workgroup collaboration. If such products are to act as evidence of the business activities that created them, particular attention must be paid to what is stored, and to what contextual and other metadata is stored with them to enable them to be evidence. Alternatively, document management packages can be used as repositories of past information enabling re-use of accumulated business precedent. Ideally both capacities are combined. We should be striving for systems in which documents are stored as authoritative records reflecting business activity which also enable extensive re-use of the data contained in them.

It seems clear that all document management packages in the marketplace aspire to be more than being clever systems to facilitate work. They are all increasingly heading to the functionality of records systems, but are doing so by reinventing wheels and by trial and error methods. The demands of organisations are driving document management packages to include functionality which address issues such as security, authenticity and reliability.

If documents created and managed in document management systems are to act as evidence of business activity and transactions, a ‘fixed’ document reflecting the business activity needs to be captured at some point. As much as possible we would like to be able to automate this and make it happen seamlessly without the creator or user having to leave the business process and consciously stop to jump to a recordkeeping process. We need to establish rules embedded in software which facilitate this capture of evidence.

There is a growing consensus that the process of ‘fixing’ should happens at the point that documents are communicated outside the context of their immediate creation. Until something is communicated no subsequent action can be informed by it - no business action takes place. Communication can be the dissemination or routing of documents beyond the workgroup. It can also encompass access to the document from outside the workgroup, whether to glean information or to incorporate some parts of the document into a subsequent business transaction.

Defining the boundaries of workgroups becomes key to establishing when a ‘fixed’ record should be captured. Establishing appropriate communication boundaries depends upon the activity the workgroup is engaged in. Hard rules cannot be established to cover all workgroups, as activities are so diverse and so embedded in the context of the particular business. Most organisations do not have a business need to keep working documents once the finalised document is produced, authorised and despatched. However, this is a business context specific decision. In legal firms (from whence much document management software developed), the product of the office is the document. Each iteration of a draft does have business significance - knowing which party added or deleted phrases or what redrafting was done in response to comments can become a significant business transaction. Similarly in offices involved with drafting legislation, complex agreements, contracts etc, the requirement to manage the document development process can be critical. However these are not the circumstances facing most organisations.

Functional analysis of the business determines where the boundaries are to be established. This analysis is not based solely upon what workgroup performed which task, but is more critically based on what the organisation does - its functions and activities. Workgroups are subordinate to the business activity. They are transitory formations able to be formed and reformed with great flexibility to enable the work to take place.

The linkage between workgroups and activity they are empowered to perform, the accountabilities and ‘competence’ attributed to workgroups, require specification to establish appropriate boundaries for records creation. Document management packages do not have these capacities. Focussing on the individual rather than the business activity, users are given certain access rights. Access rights tend to be ascribed to individual documents. Linking access rights to individuals is not the same as documenting the workgroup within business activity. Workflow packages, with their primary emphasis on business activity rather than the product of that activity, offer the greatest capacity to document which workgroups are tasked with what specific business activity at what time.

Document management systems focus on the user friendly identification of individuals by name. This is usually linked to a table specifying organisational placement and position. It is specification of the position occupied at the time of creating documents which carries with it knowledge of the ‘competence’ or accountability delegated to that individual. This in turn affects the reliability of the document. Document management systems manage the link between individual and position, but do not maintain an organisational history of what position was occupied at what time. Rather, the matrix of individual and position is updated to hold only the most current information about occupants of particular positions. Documents still linked to the individual may have been created when that person held quite different positions and responsibilities. Failure to maintain such historical tables compromises the meaning which can be attributed to documents as appropriate evidence of business activity.

At present, workflow packages provide the capacity to ‘portray’ the organisation according to business process. Even here, a recordkeeping perspective on the business functions and activities needs to be read against the business process. Business processes are aimed at providing a product-focussed outcome, defining in business-sensitive ways the best and most efficient way of achieving the specified outcome. In response to changing business needs the ways of performing tasks and undertaking business activities are altered. An underlying agreement on what the functions and activities are is the base on which workflow packages are enabled. Workflow provides us with the capacity to link processes and their documentary products directly into the business functions and activities.

Workflow packages are increasingly being bolted together with document management packages to enable the transaction of business and to manage the by-products of that business - the document. Using such enhanced functionality ‘traps’ can be located in various positions between software applications which automatically capture a locked document or record. The positioning of such traps will vary according to the business activity and the communications boundaries established at implementation. Issues such as capturing documents prior to encryption when leaving the organisation and capturing them after decryption on receipt, but recording what encryption was used, will form important components of where to implement the traps.

At present, there are (hopefully transitional) problems with such integration between workflow and document management packages. These problems are currently inhibiting efficient capture of data needed to surround a document to enable it to serve as evidence of business activity.

Metadata for records

Embedding the document within its business context is the most critical purpose of metadata required to ensure that documents will act as records. There are increasingly sophisticated specifications emerging which define what other data needs to accompany the actual content of documents. Here the recordkeeping professionals are joining forces with other information professionals to specify document metadata. Collaborative projects such as metadata specifications of the Dublin and Warwick core metadata models are defining item level metadata.

Document management packages overflow with capabilities to define metadata accompanying documents. The capacity to add extra descriptive boxes to document profiling screens seems infinitely extendable. However, just adding extra fields does not ensure that the document is surrounded with the appropriate metadata to ensure that documents act as evidence of business activity. Leaving selection of fields which comprise document metadata in the hands of users to implement without defining the minimum standards is abrogating responsibility in this area. The defence that the software is capable, but the implementation deficient, will possibly be true, but will compromise the claims of document management systems to manage evidence of business activity.

Systems architecture

Implicit in the on-going discussion of metadata specifications is the concept that the record consists as much of its metadata as of its content. Issues of systems architecture become crucial to the capability of various software packages to manage records as both content and context.

The meaning of a record is conveyed as much by its context as its content. Knowing who created it, what authority they had, when it was created, who saw the record, what subsequent use it was put to, whether it was relied upon etc provides a layer of meaning as significant as the actual content of the document. The two aspects - the metadata and the content - need to be managed over time as one entity, inextricably linked, in order to provide evidence of action. Without context, the reliability and authenticity of a record cannot be established. Lack of reliable and authentic records inhibits all future uses of the documentary base that an organisation may wish to exploit.

How software packages manage the links between the content and the context is becoming crucial. At present the metadata is managed in software such as document management systems and the content is managed as data or objects linked to contextual metatdata. The object oriented environment holds out seductive prospects of managing content and context as a single entity. Metadata encapsulated objects which embed document content within an ‘envelope’ of metadata able to be managed as one entity seems an architecture particularly suited for records. Other implementation environments include relational databases with linkages ensured by using hashing or other security techniques. The use of pointers between document object and document management software is a fragile link, not robust enough to sustain over time through software system migration.

How document management packages deal with such architectural issues will become an important discriminating point for recordkeeping evaluation.

Item level control

Increasingly we manage records at item level. This is the forte of document management packages. It is a world which harks back to the nineteenth century where records systems focussed on individual document item level. Back then, the volume of communications was much less and the availability of clerical staff supporting such activities was much greater. Transaction-oriented computer systems enable us to regain the control of items which was economically not feasible from the early decades of the C20. The precedents of C19 systems have not in the main been incorporated into today’s document management systems. Nor have the best features of the C20 records practices.

One element lacking in most document management packages is an appropriate capacity to link documents with other documents created or used in the same business activity. This linkage of documents provides the trail vital to reconstruction of events. In the paper world this linkage happened with the grouping of papers into a physical file - labelled with a title which hopefully reflected the activity documented within. Both inward and outward correspondence related to a particular activity was handled in one physical grouping. Tracing the history of an activity was relatively easy, presuming the records system operated well.

Document management systems primarily manage the internally generated records. Imaging systems convert paper flowing into an organisation into digital form. Linked to document management software, both inwards and outwards correspondence can be managed within the document management system. Alternatively, the documents flowing inwards are managed as paper. Until we reach the stage of total electronic communication, document management systems in themselves do not manage the whole documentary base. Sophisticated linkages need to be established between systems to enable the full development of a transaction to be recorded and recalled. Tailoring of document management systems to incorporate management of paper documents is possible. One such tailoring is PaperDocs, an add on to PC Docs.

Here again, workflow technologies with their process focus, promise far greater capacity to manage the linkages necessary for records management than do document management systems.

Storage and Management of Documents

All document management packages possess the capacity to store and manage documents. However, the storage and management features of most document management products are geared towards the technology - freeing the bits and bytes, effectively managing server space - rather than managing storage from the perspective of the business activity. Most document management packages allow users to specify what period of time the document should be kept for. Such ad hoc decisions are not linked to the organisational recordkeeping framework which is required to appropriately allocate and manage such functions.

The organisational recordkeeping framework looks to the ‘warrants’ mandating the conduct of affairs - the legislation, best practices, standards etc applicable to various business contexts. In such analyses the rationale for creating and maintaining documents are compared to externally imposed mandates and standards, and measured against internal requirements for accountability. Risk management techniques applied to business functions and activities contribute to determining what records should be kept and for how long. Implemention of these decisions can be automated as is evident in the Australian developed records management packages. This allows organisational rules for record retention to be applied as rules tables within software packages, minimising the need for individuals to make ad-hoc decisions about individual transactions or documents. Such functionality is not currently available in most document management packages.

Similarly the transaction history needs to be documented after the record has been stored. Who accessed this record? What subsequent transactions is it linked to? Most systems manage this as an audit or transaction log. As indicated earlier, this information is actually an integral part of the record and is critical to the acceptance of the record as evidence. Transaction logs and audit logs appropriately store information needed to reconstruct events in the event of system failure. To entrust crucial recordkeeping information to such ephemeral processes is to miss the point completely.

Information retrieval

Document management currently features powerful information retrieval capacity. This is one of its greatest strengths. Information retrieval is a business driven requirement enabling access to systems which store data, documents and records. The information retrieval capacity does not have to be part of the recordkeeping software. It seems that the rapid adoption of intranets and internet technology with their superior information retrieval tools may supercede information retrieval in specific applications. Delphi Consulting Group reported in November 1996 that 61% of the companies it surveyed are currently using intranets and 29% are actively evaluating them. Deals involving the internet search engines are some of the most dynamic business ventures being reported in the press. Using widely available information retrieval tools, such as Lycos or Alta Vista, capable of searching across applications bring the added benefit of a single user interface. Will document management systems need to duplicate the internet/intranet information retrieval packages?

The problems with many information retrieval software packages are well known - we are all drowning in a sea of stuff retrieved. The research project ‘Dying for Information’ reported in The Australian ‘found that one in four managers now claim to suffer from stress induced by the amount of information they are expected to assimilate into their working lives’. How much of this overload can be attributed to retrieving information out of context? So much of the meaning we attribute to information is based on the context it is presented in: we evaluate the authority of the source, the timeliness, the process which gathered it, the activity it represents.

Defining appropriate context and relationships between organisations, individuals, business functions and activities and the records they produce is a key skill of recordkeepers. Focus on retrieving information content, even words in context within the information content, seems to be leading us down the overload trail. What might result if retrieval tools concentrated on contextual relationship searching? Tools which deliver information in the context of its creation and focus on the searching of the context prior to the searching of content may produce different results. Information retrieval systems focussing on context will be particularly relevant when accessing the internally generated records of an organisation.

Attribution and exploitation of contextual elements surrounding information is a key value-adding component which is underutilised at present. The desire to make better use and greater re-use of the information generated within organisations and held in a variety of applications is evident in concepts such as ‘knowledge management’ or at a technique level ‘data-mining’. Without a complete contextual understanding of the information retrieved how can we be sure that it is accurate, timely, authoritative, reliable? Organisations cannot afford to base their actions on dubious or untrustworthy information created, received, or kept out of context.

Recordkeepers are also skilled at managing meaning over time. Records are time bound. They live in perpetuity with the contextual data allocated to them at the time of their creation. They reflect business activities which are similarly time bound. Language and meaning change over time. To ensure the connection between the vocabulary of today and the vocabulary of the past is a skill which recordkeepers manage as a part of the management of the relationships between organisation, function and record. We have underplayed this skill and others have seen it as something only relevant when retrieving records across large periods of time. But the skill is no less relevant to managing terminology in the current organisation: for example, notions of equal employment opportunity are relatively new but preceding concepts contain much information on the subject. Phrases such as equal pay, women in the workforce, married women employees will probably contain information that now would be classified under the rubric EEO. Similarly repetitive strain injury, a term popular in the 1980s, was referred to as occupational overuse syndrome in Queensland. The information superhighway is a concept introduced in the last 3 years, but now probably superceded by information infrastructure, following the Clinton Administration’s initiatives.

In the electronic environment we can no longer browse in ways possible with paper records. Item level control precludes finding earlier material grouped with material named according to the most recent term. The research focus on information retrieval in electronic environments highlights the importance of enhanced techniques in the electronic environment. Language can change very rapidly. Maintaining relationships between linked concepts over time in order to retrieve a full picture is a complex task to which recordkeepers have much to contribute.

Electronic documents accessible via internet or intranet technology will lead to new access regimes for organisations. There is the potential to enable access to organisation-specific information to a much broader audience. Concepts of openness, freedom of information, exchange of information, trans-border data flows, security, privacy, and confidentiality are now played out in a different dimension. Organisations will have to implement new rules on access to documents. Similarly there is a growing need to protect information extracted from an organisation - to ensure that it is not taken out of context, that its authorised status is not abused, that information piracy is minimised. This protection depends, in part, on controls implemented by organisations to ‘stamp’ their documents authentic, to ‘authorise’ certain uses and to manage those documents and permissions beyond the existing organisational boundaries. Management of access regimes is another skill which recordkeepers can bring to organisations and software developers engaged in thinking through the ramifications of transactions in cyberspace.

Conclusion

Document management systems offer powerful, but expensive, tools to organisations. Increasingly they imply that they are capturing and managing the documents produced in business activities as evidence of those activities. The software developers are enhancing the product capacity to meet the perceived demands of organisations, but have not recognised the additional requirements they must meet if their products are to deliver on implicit promises to be electronic records management packages. In meeting these challenges, the recordkeeping profession has a great contribution to make to product development and to organisations seeking reliable evidence.