The Problem

You have a well used application that has served the organisation well for many years but now it needs to be updated. Perhaps, through a change in the business structure a choice has to be made between one system or another. Sometimes the cost of maintenance of the original hardware is forcing a change,  or it is new government regulation that requires different methods to be used.
The data cannot be simply archived as you must keep it for 10 years and the latest information is only a year or so old.  Although some of it is static information and may only be required once or twice, other information is vital and must be transferred in a sensible way into the new system. The dilemma is having done that do you then maintain the existing hardware and software or try to extract it in some way. Many systems have tools and reports that can extract data but how can you be sure nothing that may be needed later is left behind.

It is these problems and the issues associated with legacy systems and the introduction of new work processes that we can help you solve.

The Solution

Archiving

With our experience of different methods of storing data we have developed techniques for extracting data from legacy systems quickly, reliably and accurately.

However before starting work it is vital to establish exactly what importance is to be placed on sections of the data.

This stage should not be rushed. Many of our clients have turned intractable problems into relatively easy ones by using our methodical approach.

It is impossible to say, before examining the work process in detail, what is the best course however we typically find that with some administrative effort the problem can be cut down to a more manageable size. In some cases it has proved sufficient to print off a report for say a couple of hundred cases leaving the majority to be dealt with by other means. Perhaps with some extra staff working on an issue the number of exceptions can be reduced to a negligible amount.
During this early stage we tend to identify mismatches between the legacy system and the one to be used in the future. It is important to be sure what the information is used for. It is often a typical error to assume a field that may be something called notes can simply be dropped, it is often surprising to find that the key to any future query is actually contained in this data field.
As a fundamental archiving principle we preserve all the data and transfer it to an intermediate data store for further processing.
If it is found that some data can be truncated or left behind then there is always a way back to the original data if this assumption proves in practice to be wrong.

If required we then build an interface, either via a desktop application or web browser to allow the archived data to be readily viewed in a form they are familiar with from the legacy system. This is read only of course, but provides staff that need it with familiar context in which to view the data.

Extraction of the data

We have accumulated many years of experience extracting data from a variety of hardware platforms and software applications. The techniques and tools used by our team depend on several factors, all aiming to produce a reliable consistent result.

The objective

When we extract data from a system we aim to make the process repeatable. We design the software and tune its performance to run the process many times to allow changes in the existing system to be seen in the new system as soon as possible. If we run an extraction regularly, we need the process to be reliable, consistent and need little human intervention.

In an ideal situation, data would be extracted from the existing system at night. The data would then be converted using the rules specified in the parameter mapping process and then loaded into the new system. This process has proved to have been highly successful.

What are the advantages of this approach?

There are many advantages to regularly 'refreshing' the data in the target system. The main ones are:

* The ability to see the effects of recent changes
* The ability to quickly identify issues
* The ability to use the old system to make corrections.   This means existing staff can correct issues in a familiar system
* The ability to see new parameter values as they arise
* The ability to keep working with the existing system for much longer

* Changes to resolve issues can often be made in the existing system where users are comfortable and familiar.

Parameter Mapping

What is a parameter?

A parameter is simply a piece of information that can be associated with another piece of information. For example, in the context of a housing management system, a property might have 'heating type' parameter. This parameter would enable the type of heating to be specified. A property might have a heating type of 'Gas fired-Central Heating', 'Coal Fire', 'Night storage heaters' or many others.  

It is also possible that a parameter in the new system may have been a combination of more than one parameter in the original system perhaps with some complicated rules as well.

Why do parameters need mapping?

When moving from one software application to another it is rare to have everything in the old system look identical in the new system. A new system will usually work in new and improved ways and will need the information it uses to be set up differently to allow it to work effectively. The mapping process matches up information from the original system to allow it to be placed into the new system.

How does it work?

Parameter mapping is mainly a manual task. We have techniques that make the task as easy as possible by using a process that is consistent for each parameter and has been proven on many projects over the years. From the outset we identify all the places that a particular parameter is used in the existing system. Next, a list of all the values used are found along with a list of all the values that are available in the destination system. These values are then displayed on a web page or on a spreadsheet along with any mapping rules that have already been defined.

The spreadsheet/web page is generated as part of the Data Extraction and transformation process and will reflect any changes made to the existing system and destination system. The spreadsheet is usually examined by each member of staff who is directly involved in part of the system that the parameter would be used in. We believe the more people that look at the sheet, the more likely issues will be spotted sooner and resolved more easily.

What effort is usually involved?

Usually each spreadsheet simply needs someone to decide what existing value needs to be made to match up with the new value. Each new value can be used by more than one original value. There is no need for all new value to be used.

Sometimes values are being used that no-one is aware of. It is common for us to be told that "there are no records using that value, we stopped using it years ago." We often prove that the value is used and then either it needs to be mapped, or changed in the source system. If it is changed in the existing system, the Data Extraction process will register the change the next time it is run and the parameter value will disappear from the sheet.

Data Transfer