Capturing Electronic Transactional Evidence: The Future
Barbara Reed

KEYNOTE ADDRESS, RMAA Conference, e-Business Transactions: Providing Accountability through Effective Recordkeeping, March 2000

There is really no question about the future of capturing electronic transactions - it will happen, but how and who will be doing it are different questions. Managing transactions is the life blood of recordkeeping. It is what we do. On the business front, there is no reason to suppose that the fundamentals principles of business transactions will change with the introduction of new and different technology. Indeed the recent passage of the Electronic Transaction Act (in force from 14th March 2000) reinforces that point. Electronic transactions are the stuff of the future and integrating them into our business systems is imperative - not an optional add on to doing business, but imperative - for the survival of all businesses in the electronic world.

Discussing recordkeeping and the requirements for capturing electronic transactional evidence could involve considering the extension of some initiatives that are being discussed and investigated in our records world. Such discussion would lead us down the arcane paths of:

• Distributed computing systems and protocols
• Metadata standards for recordkeeping
• Tools to underpin those recordkeeping standards
• The slow pace of change of our current records packages (recent much lauded software releases in some packages have just worked out how to deal with electronic mail, a real problem for at least the last 6 years! - and even then one can question the approaches being championed)
• Mindsets, standards of education, competency and innovation in the records profession and their capacity to meet the challenges that lie ahead
• The unenviable position of software developers in records markets needing to straddle organisations at various stages of technological development from those working almost entirely in paper, right across to those that are almost entirely electronic transaction driven, with all the problems that such a diversified technologically-aware market provides.

Rather than do this in a serious-minded manner, I've chosen to play with some stories and from three views of the story to explore what might be, in the future.

Its 2015 (or any time in the nearish future). Government has been subject to almost continuous review of its core role and how it delivers essential services to its constituents. While active in the promotion of electronic service delivery and the stimulation of the e-conomy for over 25 years, there was a much slower uptake of the radical nature of the organisational changes needed to transform the bricks and mortar mindset to the clicks and mortar reality of today. Now we find that the rhetoric does reflect the reality and electronic services are predominant, with uniform front office systems dealing with the client interface across all erstwhile independent department structures. The Governet project, successfully implemented in the early part of the century, with its single search interface over three tiers of government has been replaced by a far more sophisticated portal allowing the public or clients of government services to integrate services from government to private companies as suits their specific-instance needs. Much publicised through the media, this intelligent portal, known as FPOC (first port of call), is the first point of service for all government-citizen interaction.

A client view of the scenario

Tran s'Action is an Australian born son of East Timorese refugees who arrived during the turmoil of the last year of the previous century. He needs to prove his compliance with eligibility criteria set up by 'Innovative Business', the government support network now awarding innovation grants to Australian businesses supporting themselves internationally by electronic transactions. To make the application Tran needs proof of business experience of the family run enterprise over the past 10 years and proof of citizenship of all claimants. As the business Tran runs is jointly owned by Tran and his parents, obtaining this information from authorised sources involves interaction with a number of government agencies. It involves delving into the circumstances of his parents arrival, their date of naturalisation, citizenship and his own citizenship records. It also requires independent proof of the business conduct over that period of time, and the evolution of the business to the innovative organisation it is today.

Tran logs onto the Government Portal to find out how to do this complex set of transactions. The FPOC allocates him a minder, interactively connected to the web-site who converses with him (in a mixture of keystroke text and verbal interactions) as he explains the issues. The Minder is actually a composite entity fronted by a senior analyst located in Tasmania, the hub of all government service delivery. Together they piece together a strategy for dealing with Tran's query, establishing an ad hoc set of interactions across the various government services of refugee administration, immigration, citizenship, and births deaths and marriages that he will need to prove the personal identification details required. A second template is established to query the taxation system and business returns of the Australian Companies and Securities Commission to provide evidence of the business operations and compliance over the years. Tran is somewhat surprised at this personalised level of service and as he didn't expect it, having heard nothing but the usual bleats about government inactivity from his colleagues. He is not able to deal with the processing of these templates immediately, so the service templates are saved under a public query system which issues Tran an identification number and transaction number which Tran can then access and invoke at his convenience.

Tran returns to his templates the next day, equipped with more time at his disposal. As a first step to following through any of the transactions, he needs to establish his digital persona. The issue of digital persona is still a vexed issue for the government. Because of the problems with the privacy regime over the years, which have still not been resolved, almost everyone transacts business through a digital alias. These digital aliases need to be confirmed as genuine and authorised before transactions with government can be actioned. Tran authorises IRE (the Identification Registration Enterprise) to verify his digital persona for the purpose of 2 specific template executions, citing the transaction numbers allocated to him in the public query system. A message from the IRE lets him know that this has been approved. Tran is authorised and is permitted to the next stage of the sequence which is contact with the refugee section of the old Department of Immigration and Ethnic Affairs. This interface requires more information and authorisation to release information about personal details of his parents. Tran obtains a verbal release from both his parents which is encoded into MP3 for digital transmission to the Department. Again there is a validation process, through IRE, while the voice prints are matched to the identity records to check that they are genuine. The transaction is approved to continue. Some days later, Tran receives an encrypted message from the Department with verified attachments from the Department's records system containing the details he needs - scanned paper records of arrivals, transcripts of their refugee interviews and authorisations to his parents to remain in Australia, all dating from early in the century. Authorised copies, with digital verification from the department, of their citizenship certificates are also included. Meanwhile, the template has transferred him through to the State Department responsible for births, deaths and marriages, in reality, the controlling parent body of the IRE, to pursue his request for his own birth certificate. The identity checking and verification is not needed here, as the template links the transactions under one umbrella, so the earlier authorisations apply. Again, a watermarked encrypted document arrives in his email within days.

Tran, having completed the first template with all the steps he needed to do, moves into working through the second template - designed to locate authoritative evidence of the business' operations and interaction with government over the past 10 years. This template links queries and processing within the Tax Office and the ACSC. Once on track, a similar pattern of verification is the first step, not for Tran's digital persona this time, because this was inherited from the first template, but for Tran as a company director and for the company itself. At one point Tran gets lost in the system, it seems to be frozen somewhere inexplicable and expect something that Tran doesn't understand as a key to move forward - but a quick help call to his Minder, available on line for assistance, sorts this out. Again for Tran, once his direct input is established and his details submitted in text, the template operates invisibly. The Minder set up a query facility for him, which enables him to check that things are progressing and to get an estimate of the time it will take to complete the actions. Within days, the required information is returned as authorised documents - copies of annual returns and tax returns - with watermark seals of verification which prevent tampering by third parties, are returned.

Tran is one happy customer. He can't believe how responsive the systems were and how effective the templates established by his Minder had been. He now completes his Business Innovation application and attaches the verified documents as required and submits it to the local business enterprise body.

A Services view

FPOC creates a unified front across all government resources and people to meet the specific customer interaction. The Mentor, which appears to Tran as one single person, is really a composite made up of a virtual team brought together on the fly to construct the paths and information flows needed to satisfy the query. The team is drawn together for a variety of subject specialist knowledge. Selected from a sophisticated knowledge registry, displaying as a directory-style interface, this registry works across knowledge specialisations of government processes. These specialists in government processes have evolved from the front line staff of the Call Centres - now doing far more sophisticated knowledge mapping and process mapping quite far removed from the original telephone directory services from which they evolved.

The team is headed by a senior specialist, a Template Architect, who is charged with eliciting from Tran exactly what his requirements are. Using the array of information resources which would be familiar as descendants of the old information resource discovery sets, enabled by AGLS metadata, working across government services, functions and people, the Template Architect coordinates the virtual team as they construct the specific templates needed to satisfy Tran's query.

The software that supports the process derives from older process mapping software, much enhanced with graphical icons enabling quick building and bundling of processes and information flows. At various times, automated checking agents are invoked to ensure that there are no obvious process gaps and that the links have been constructed correctly. The process of building the template, the talking through of the problem and the mapping of the interaction points with various independent parts of various administrations are recorded through the templating software.

Where the transaction crosses responsibility lines, the template explicitly nominates the various agency responsible for separate parts of the stepped sequence of activity. The links are made explicitly, usually to the POS (point of service) nominated by each individual agency. From there, agency specific protocols are invoked to route the questions and requirements through to responsible officers or systems. At every point along the sequence, an individual is nominated as responsible for taking action or ensuring that the appropriate action is done.

As much of the interaction as possible is done by data mining agents. Initially invoked from the POs controls they identifying records and information systems needing to be queried. This is done by searching the functional descriptions of what each system does and the types of data they contain. Such information resource directories were tried in old paper systems and in early days of the electronic era, but at that time were dismissed as unnecessary overheads as they only statically recorded information for asset management and inventory purposes. Now there is an immediate client related, business use for this type of systems, their uptake has been an important part of enabling the integration and utilisation of data from disparate sources including legacy systems.

A major part of the role of the front office - both the overarching FPOC and the agency specific Point of Service - is to do the validation and authorisations. Individuals on the web have been pesky to identify since early days of the web, operating through multiple net persona. The population as a whole has recognised these problems, with the realisation that personal details were available all over the world wide web and could be patched together with ease by marketers, advertising companies and others capable of exploiting the ubiquitous cookie software. The accepted alternative, is government controlled person registration, with the protections of privacy and security mandated by law. Dogged by controversy similar to the debates from the late twentieth century over Medibank numbers and Australia Card, the uncontrolled reality of abuse provided the approval of citizenry to the establishment of the Identification Registration Entity (IRE), an extension of the State based Registries of Births Deaths and Marriages.

Individuals and companies who interact with the government (and other major institutions) on a regular basis have voluntarily signed up with an identity registration entity. The IRE operates according to standards on checking and validating that are much higher than mandated in order to keep faith with the generally still skeptical public. The IRE registers net identities against real people. Individuals must go through biometric identification and a 100 point check (similar but much more stringent to those controls imposed upon opening bank accounts in the 1990s.) Given the propensity of net transactors to use different identities to undertake different types of business, the registry enables multiple net identities to be registered against one real person. Other transactional entities are registered also. The ABN scheme introduced as a part of the old GST taxation regime provided an easy platform onto which to piggy back the authoritative identification of organisations. Individuals and organisations can, once registered, request verification of any of their net entities to those enquirers nominated as part of the transaction to be undertaken.

The Template Architect assigns specific responsibility to individual client services operators from the various agencies for the performance of the templates. The template system routinely reports progress through the sequence of steps identified, and graphical timelines can provide visual displays of how far through the process Tran or any other client is. Standards for response times and overall transaction times have been built into the system enabling re-routing of transactions stuck in some unexpected queues. Supervisory mechanisms can query the entirity of customer interaction through FPOC. Individual agencies can be analysed to assess performance standards and systems capacities. This is a sensitive performance indicator to the Commonwealth government and politicians , particularly Ministers, take these performance measures very seriously, as they are a front line between the citizenry and the bureaucracy.

A records view

All of the interactions between Tran, the client and the various people involved in dealing with his sequence of transactions have been documented by clever capture mechanisms incorporated into the template. As the template was being developed, the Template Architect indicated the points at which a record was to be created and captured. Once identified, these points invoke a specialist piece of componentised programming, similar to the old Java applets. This applet is invoked as the step is completed, bundling separate computer transactions into a business transaction, prior to the template rolling on to another step in the sequence. At these nominated points, the specialised program known as the CapBot is pulled down from the Recordkeeping Authority's application server. The Recordkeeping Authority establishes standards, verifies compliant software according to the specialist recordkeeping processes needed by the agencies (or others) at a specific time and acts as a broker for delivering specific compliant process-oriented applications. These mini-applications are issued on demand under an operating licence to requesting organisations. They involve a small, per transaction, fee, payable to the various software design companies under contract for the specific functionality at any one time.

The CapBot is pre-programmed to conform to the various standards established by the Recordkeeping Authority working in conjunction with its State based counterparts. For the 'FPOC' site, these currently include all specifications to do with the transactional metadata needed to document the specific transaction, a segment that grabs contextual information from the web source itself to locate the accountability points, and the storage format standards currently mandated. These various standards and specifications have changed periodically as the Recordkeeping Authority updates its requirements based on the latest available technological innovations and standards in the distributed electronic world.

CapBot is essentially a clever metadata collection agent. It locates information corresponding to its pre-defined metadata elements - information and data which may reside in many different places in agency systems and incorporates them into a recordkeeping metadata template. Amongst many other things, they record:

• the transaction number,
• creator identification (including all people who contributed to the composite entity, the Mentor),
• the security coding including all user identification codes allocated to Tran to allow him access from outside the system and to control unauthorised access by third parties,
• what activities were being done (customer service - refugee status, immigration, citizenship, taxation record, company registration),
• details of the style sheets which displayed the graphical representation,
• details of the workflow template creation system,
• formats and scripting protocols,
• time, dates etc.

The CapBot metadata routine embodies identification of the schemes used to define the metadata elements at various times as standards evolve. Each one of these metadata elements is defined according to schemes which are universal in their reach. The agent registration identification, for example, is inherited from the IRE identification scheme. This scheme was already in place and designed to work as a unique identifier across all digital environments. The validity of this approach, once reviled as being over the top, is now understood to have business value as the physical location of any transaction record is totally immaterial to the creating agency and their creation systems. Designated stores can exist in any computer device continuously linked to the net. Indeed it would be very rare now to find transactional records all physically situated in one system. Records, like so much else in this digital distributed world, are virtual representations of transaction, relying upon the metadata appended and embedded to provide the identification, authentication, and intellectual property rights required to access the content.

The CapBot renders these metadata tags into ZML standards, the current flavour of the month net metadata language, conforming to RDF specifications. The actual business transactions are bundled with their metadata tags and converted into the storage standard currently mandated by the Recordkeeping Authority. The metadata is persistently linked to the record object itself through inviolable binding protocols.

The CapBot is invoked at the predefined points of the action template established by the Template Architect. Records are bundled together within FPOC, but also maintained as discrete entities within each of the agencies involved with the actioning of the sequence steps. This apparent redundancy was resisted for some time by information technology specialists, but the business need to maintain accountable transactions at the point of action, rather than only as an aggregate FPOC transaction, was clear to the senior executives involved with reporting on accountability points.

The CapBot operates across all systems, creating authoritative records of transactions on the fly. Old legacy systems with automated control systems are able to be integrated using the data mining agents, which query specific systems protocols and represent the record. At this point, unless wholescale digital scanning systems were implemented, the paper itself does need retrieval for scanning into the end product required by the client. Watermarking and other authentication mechanisms involved in verifying the authority and integrity of the migrated record are part of the representation issues addressed by individual agency systems.

Once captured, records are sent off to find a storage location, reporting their location using persistent digital addresses. The Recordkeeping Authority has approved a number of other records-process oriented segment applications. They include periodic maintenance procedures to destroy records objects passed the time nominated for retention. Another quite sophisticated set of applications deals with migrating electronic records into new formats as technology standards evolve. This includes translation of both the storage formats - early records were stored in PDF and other open systems. A transfer application formally documents changes in the responsibility for records with the demise of specific agencies or with the passage of time.

A Step Back

Let us abandon the fiction at this point. It is one, contestable and entirely optimistic, view of what the future might hold for mechanisms and technologies for capturing electronic transactions. Lets explore some of the things that are inherent in the weaving of the stories.

1. The centrality of the customer

The experience from the electronic commerce world is that the conduct of business is being re-personalised. Individuals able to interact with far greater freedom with organisations are requiring personalised service, rather than service delivery geared at operations at a mass scale. This involves a substantial re-think of the way services are delivered and the way systems are designed to facilitate delivery. While the examples in my story involved government based functions, the same shift in thinking is most pronounced in the dot.com companies emerging in private enterprise.

2. The continuing involvement of real people

The story weaves the interaction of clients with real people as they are needed to establish processes and facilitate remedial action as needed. Why introduce real people into processes designed to be automatic? Early ecommerce implementers removed this personal contact from systems designed to support customers but recent experience suggests that this may backfired somewhat. While automation is all well and fine and can deliver things differently, for non-standard, one-off or ad hoc queries it will never be feasible to establish totally automated systems. Even the routine transactions suited to automation have hit their trouble spots - this can be seen in the outrage of electronic shoppers at Christmas left with no recourse as their orders were unable to be fulfilled by the Christmas delivery times promised and in some cases, orders unable to be cancelled. Some commentators have raised fears that such initial negative experiences may have set back the adoption of ecommerce in the broader population.

3. Records of transactions will continue to be needed

The story, in fact, revolves around records - the need to provide authorised evidence of action relating to Tran and his parents and company. The need for accountability, places to go to check up what is happening or to trace how occurred, will be essential in the digital world in exactly the same way it has been needed for conducting all business over the past millenia. In the story, a record or documentary path in the form of the action template actually drives the interaction between the client and the various participant agencies. The scripting of the interaction is creating a workflow, a continuing essential component in records formation - intrinsic and essential linking of action and the documentation of action at the time it is taking place.

4. A clear understanding of what records are and how they are constituted

Much reviled or observed with amazement by many of our information colleagues, recordkeepers undertook a deeply introspective look at what records were as the digital revolution approached us during the 1980s and 1990s. This fundamental reevaluation of the constitution and role of records has paid dividends over the last 5 years, as we have been able to define with clarity and certainty what records are and how they need to be protected. The constancy of records requirements transcend individual technologies which change with the wind. Technologies are not static and will not be static for the foreseeable future. Records on the other hand are constant.

Australians have been prominent in the re-definition of records functionality and requirements. Our coherent theoretical basis has united many collaborative community projects resulting in strongly adopted standards on records management which are now being translated into the international standards environment.

5. A changing software environment with distributed systems

The computing paradigm around us is changing. The influence of the internet is pervasive. New ways of doing things are the norm. Start up companies have an advantage in adopting these technology approaches from scratch rather than having to interact with existing systems and ways of doing things. However far from the reality the scenario outlined in my story is, some things seem clear:

Applications will be leased rather than owned: Microsoft in its post anti-trust state is proposing to become an applications provider. Programs will be invoked as needed for a different type of pricing arrangement than those we know now. Some will be free.

Applications are being bundled both as single applications and as componentised parts: Clever implementers will be able to bolt various program bits together as the technology evolves. While the volatile on-line environment is busy churning at the moment with big fish eating little fish for exorbitant amounts of money, for the consumer the disaggregation of components is equally as likely as the aggregation into monolithic systems.

6. The mainstreaming of middleware and agent based technologies

Agents have been much discussed in the software profession during the 1990s. Early applications of artificial intelligence in the form of personalised agents were to be seen in products like Ask Jeeves. It seems to me that these things are retreating as agent based protocols become more sophisticated and are involved at multiple levels and multiple points in facilitating on line actions. The embedding of some operational knowledge into bits of code is what I interpret as agent software and it is a major component of the middleware and resolution services which mediate between client and action embodied in any number of forms. These agent based technologies are set to grow. In my story these technologies drive much of the authorisation, verification processes outlined.

7. Data mining and data visualisation

At present data mining is much talked up as means of extracting useable data from a variety of disparate corporate sources. But I think it is still an application looking for a coherent business purpose. One of the problems with traditional data mining implementations was their complete ignorance of context. Data from a variety of organisational systems were aggregated together with little understanding about where the data came from, who created it, whether it was authoritative etc. Similarly data mining has tended to mean the construction of a large system. Rather than pursue this type of implementation of data mining techniques, I suspect we need to build from these techniques to understand the contextual elements and their importance for interpretation. With nifty software protocols, we can unify the results of data mining without creating an additional infrastructure. Data visualisation, or the capacity to interpret data through graphical representations must surely increase in popularity with our icon oriented, visually aware population of users. Things that make information digestable and understandable through a variety of techniques will surely grow.

8. The evolution of single entry points

At present in the online environment we are presented with a relatively unsophisticated view of what is possible. Information is regarded as a passive resource to be delivered as a blob or entity to the end user. Finding or retrieving the resource is all pervasive as a design driver. Now we are moving towards electronic service delivery, some of this passive notion of information is being challenged. Focussing on accessibility, the resource discovery community has only gradually come to incorporate the capacity for doing something within the representation of information resources. Portals will get intelligent. Who knows what form this will take, but the story plays with what an intelligent portal might do.

9. Primacy of metadata

The scenario highlighted the role of metadata in telling the records part of the story - but the importance of metadata and standardising representations of metadata is already deeply embedded in all business systems, maybe just not known by that seeming buzz word. Metadata has always been a key component of our business systems, including records systems. However, at one level, this emphasis on metadata indicates a radical shift. It moves the emphasis from systems onto the stuff within the systems. It seeks to future proof information content and this is a necessary corollary to realising that business systems applications are essentially transient things. Applications need to change according to the way the business changes and use the latest and greatest techniques around. They need to be responsive to changes and new generations of software are released every 2-3 years at present. Hopefully some time in the future this will calm down a bit, but there is no indication of it at present. In such an environment, we need to place our emphasis not on the ephemeral systems, but on what they store or create. Metadata is a means of doing this.

10. Recordkeeping metadata

Understanding this role of metadata in digital environments has led to various metadata initiatives over the past 2 years. All of them owe more than a passing acknowledgement to the collaborative research project Recordkeeping Metadata Standards for Managing and Accessing Information Resources in Networked Environments Over Time for Government, Commerce, Social and Cultural Purposes. The major outcome of that project is the Australian Recordkeeping Metadata Schema (RKMS). This has been used as a foundation document for creating jurisdiction and industry sector records metadata sets, most notably in the National Archives of Australia's Records Metadata Standard for Commonwealth Agencies and the about to be released NSW State Records metadata standard. In my story, the metadata specification of the Recordkeeping Authority has been built into the CapBot to automatically generate and/or identify metadata tags.

The RKMS establishes three primary entities involved in documenting records - agents who do business, the business that is being done and the records that result from or which (increasingly) drive the automated business. It is an extensible set inheriting AGLS protocols and is a framework set rather than an implementation set. It also embodies the understanding that the metadata requirements for documenting recordkeeping over time will probably not be found in one system, but will probably reside in many business systems. Indeed this is the reality now, but we bring so much implicit knowledge to our own contexts of work that we can easily overlook how much we instinctively filter interpretation of information. In entirely digital worlds we need to make this implicit knowledge explicit.

It is this framework capacity, which enables the mapping of disparate metadata sets which have been individually tailored to specific jurisdiction, industry or organisational needs, to a common base line for interoperability of semantics, that is being actively explored at the moment. An ambitious project has been approved under the auspices of Standards Australia to evolve the RKMS into an Australian Standard to enable the persistent mapping of recordkeeping metadata elements over time.

11. Meta metadata managers

In working through the findings from the research project, it becomes clear that recordkeepers have significant experience to offer the metadata community in managing complex metadata sets. Already the RKMS has pushed the boundaries of how to identify and cite metadata tags. The relational and contingent nature of recordkeeping has pushed researchers into dealing with all types of complex relationships between people, what they do and the results of the actions, which change over time and according to roles adopted by people at various times (or even simultaneously).

With the development of the RKMS into a framework standard, a more sophisticated understanding of managing metadata sets themselves is evolving. At a minimum this involves the need to identify and date schemes for metadata elements and then provide translation protocols over time. The metadata community as a whole is coming to an understanding of context dependency and the metadata sets too are context dependent. Recordkeepers have a role in this new metadata management field. The world of managing electronic transactions as evidence of action is merely visible to us through a gap, which is currently open only a little way, but which is widening daily. We had better be ready for it. To be ready for it, we need to have some idea of where we might be going, even if the road maps need to be radically overhauled on a continuing basis. When preparing for this talk I browsed around Wired which stated: 'All commerce is e-commerce. So get used to it.' My message is not quite so blunt, but the ubiquity of electronic business is coming very fast. We'd better get used to it.