How is data understood through reverse engineering?

Reverse engineering forms an internal part of the whole software re- engineering process. To implement the reverse engineering properly in the re- engineering process one needs to understand the data through the reverse engineering.

In this article we have taken up the same topic i.e., “how is data understood through reverse engineering?” 

- When the reverse engineering is carried out, the main objective is always to recover the designs and specifications of the software system or application.

- With reverse engineering, the development is only understood and no changes are made to the software system or application. 

- For the reverse engineering process, the source of the software system or application is fed as the input. 

- If you go through the history of the reverse engineering, you will find some cases where the executable code is given as input for the reverse engineering process. 

- The reverse engineering though being in a great contrast with the re- engineering forms an internal part of it. 

- Reverse engineering serves the purpose of recovering system design and specifications in the software re- engineering process model. 

- These recovered designs and specifications are used by the engineers to understand the software system or application before they start re- organizing the whole structure.

- However, there have been cases in which the reverse engineering is not always followed by the re- engineering process. 

- There are three stages in reverse engineering process.

 1. System to be re- engineered is subjected to automated analysis.

 2. Then it is manually annotated.

 3. With the system information obtained, a whole set of new documentation is generated containing:

(a)  Program structure diagrams

(b)  Data structure diagrams and

(c)   Traceability matrices.

Activities in Reverse engineering

In reverse engineering the data is understood by the 3 basic activities that involved intense understanding efforts:

1. Understanding process: In order to understand the procedural abstractions as well as functionality analyze the source code on the level:

(a)  System

(b)  Program

(c)   Component

(d)  Statement and

(e)  Pattern

2. Understanding data: It analyzes the internal data structures and data base structure.

3. User interfaces: It analyzes the basic actions processed by the interface, system’s behavioral response to those actions and equivalency of the interfaces.

More about reverse engineering....

- The beginning of the reverse engineering process is marked with an analysis phase in which the analyzation is carried out with the help of automated tools in order to discover the structure of the software system or application. 

- But this stage itself does not suffice the purpose of recovering the whole structure. 

- Engineers then have to work on the program code and model and add the recovered information to this.

- The recovered information is maintained in form of a directed graph. 

- The engineers always make it a point to link the directed graph to the source code. 

- The directed graphs are compared with the code with the help of the information browser stores. 

- This graph helps in generating trace ability matrices and the data structures. 

- Tools that are used make it easy to navigate through the code. 

- After the complete generation of the design documentation, the information store is supplied with the additional information as a measure to re create the specifications systems. 

What is meant by Software Reengineering Process Model?

Software re- engineering process model is based up on the principles of the software re- engineering that was proposed by Chifosky in the year of 1990.  Before we discuss about the software re- engineering process model let us see what software re- engineering is.

What is Software Re-engineering?

- The software re- engineering is considered to be a process that alters a software system and thus reconstituting it in to a new form that is better than its previous form.

- The process of the re- engineering usually occurs after the software system or application has been reverse engineered. 

- The re- engineering is also carried out for the addition of the new functionalities or to correct the discovered faults and errors. 

- Sometimes this whole process is often collectively called as reverse engineering. 

But there is a subtle difference between the two! 

- The former process i.e., the reverse engineering is like an initial examination of the software system and the latter is like the subsequent modification. 

- Re- engineering of the software systems often comes in to play where there is an involvement of a legacy system. 

- In such cases the re- engineering can be considered to be a process that re- structures a legacy system in to a new system that is constituted of much better features and functionality and maintain the quality constraint.

Re- engineering is not a thing that can be achieved in an instant rather it takes a lot of time since there is a utilization of many information technological resources over the years. Therefore every organization requires a well defined strategy for carrying out the software re-engineering. 

Steps in Software re-engineering process model

The software engineering process model consists of the following steps:

1.Inventory analysis: Analysis of all the active applications listed in the inventory of an organization.

2.Document reconstruction: The software systems that are business critical are re- documented.

3. Reverse engineering

4. Code restructuring: It involves the analyzation of the source code and reviewed again.

5. Data restructuring: It begins with reverse engineering and ends with the review of all the data structures.

Scenario behind software re-engineering process

- Software re- engineering process serves the purpose of bringing up the poorly documented and poorly structured code to a standard that is maintainable and acceptable. 

- Let us see the scenario behind the need of re- engineering. 

- In a code that is poorly controlled and maintained, lies either a large number of files spread over a number of locations or consists of only a single file consisting of a very lengthy code.

- Observing these files, one could notice that the comments do not appear to have any relationship with the code itself and the associated documentation has become outdated or refers to the previous versions of that software system or application that has underwent many modifications since then which were left undocumented. 

- To say, an executable file does exists but many decisions have to be taken to run it. 

- At this point the identification of the actual objectives of the software re- engineering process becomes important. 

- There might be so many problems underlying those decisions that now can be easily addressed. 

- It is very important that before beginning with the re- engineering of the whole code that you check its correctness otherwise you will suffer a major set back if you would have been re- engineering the wrong code all the time. 

- At the end of the process the final code needs to be base-lined with the associated project log and the documentation.

- The re- engineered software system or application is brought to a well defined stage at which it can be considered to be ready for the release and also its future maintenance will be less arduous.

Data Restructuring - An activity involved in software re-engineering process model

Software re- engineering process model constitutes of the below mentioned 6 major  steps:

1. Inventory analysis
2. Documentation reconstruction
3. Reverse engineering
4. Code re- structuring
5. Data re- structuring
6. Forward engineering

In this article we have discussed about the 5^th stage of the software re- engineering process model i.e., the data re- structuring. 
Without a working code or an active flow of data a software system or application cannot be considered to be working. Therefore for the software systems and applications which have become outdated should be again brought up to the current acceptable software standards and data re- structuring is the one of the processes that can help in this regard. 


- This step of software re- engineering process model takes into consideration the restructuring of the data without making changes to the functionality of the software system or application. 
- With the re- structuring of data, the maintenance of the whole software system or application becomes so very easy. 
- The need to re- structure data arises when the data bases become too obsolete to be handled directly. 
- When the changes made to the data are confined only to the sub system, the data of the sub system needs to be re- structured. 
- The data re- structuring becomes even easier when the data re- structuring tools are easily available. 
- This step is carried out after the completion of the code re- structuring. 
- Unlike the previous re- structuring step, this step calls for a full cycle of reverse engineering. 
- The current structure of the data is dissected while defining the data models and the existing data models are run through a quality check or review test.
- So many of the legacy systems tend to make use of the global data structures and shared tables in order to save the memory space.
- Such shared data structures need to be converted in to objects like ADTs for restructuring. 

Benefits of Data Re-structuring

- One of the benefits of restructuring the data is that it certainly improves the quality of the software program or application. 
- Furthermore, it becomes easier to test and debug the software system or application along with a reduction in the efforts in the maintenance of the software and also the frustration of the developer is reduced. 
- As a bonus point, the productivity is also improved. 

Steps in Data Re-structuring

The re- structuring of the data involves the following steps:

1. Analysis of the source code of the application.
2. Redesigning of the data
3. Standardization of the data record.
4. Rationalization of the data name.
5. Translation of the file or data base.

Drawbacks of Data Restructuring

In spite of having so much plus points, the re- structuring does have some minus points which have been listed below:

1. Re- structuring does not improve the understandability of the data driven programs.
2. Re- structuring demands for heavy computations.
3. Re- structuring can lead to a loss of source code comments.
4. Restructuring does not helps with the problems with poor modularization i.e., the programs in which the components are wide spread through out the program.
5. Restructuring can lead to the loss of documentation.

From the discussion above, we can say the restructuring of data is a process that re –structures the source data in to the target data during the transformation process. 
Data re- structuring forms an integral part of the software re- engineering process model and therefore the above mentioned disadvantages of the data re- structuring are ignored. It is necessary that the data re- structuring is carried out after the code re- structuring since during the execution of the program, the code only handles the flow of the data. Therefore if the code itself has not been re- structured before re- structuring the data then the whole process does not makes any sense. 

Reverse Engineering - an activity involved in software re-engineering process model.

Software re-engineering process model is a very generic process meant to uplift the standard of the poor code that is currently not acceptable. This model is known to have 6 major stages namely:

1. Inventory analysis
2. Documentation reconstruction
3. Reverse engineering
4. Code re- structuring
5. Data re- structuring
6. Forward Engineering

This article is all about the third stage i.e., “reverse engineering” which is an important concept in itself. 

Re- engineering is usually required when the some of the sub systems of a larger software system or application need to frequently maintained. The re- engineered system is then restructured and re- documented. 

What is Reverse Engineering?

- The reverse engineering forms a very important factor in success of the re- engineering process. 

- The reverse engineering can be considered to be a process of recovering the design of the software system or application that is to be re- engineered. 

- This step involves analyzation of the software program in an effort to obtain an abstract level representation of the program that is higher than the level of the source code.  

- The software system or application is analyzed in such a way so as to understand its design and specifications and requirements. 

- Reverse engineering is an entirely individual process.

- In some cases the reverse re- engineering may be used  for specifying a software system before it is implemented again. 

- The reverse engineering process makes use of the program understanding tools like:

1. Browsers
2. Cross reference generators and so on.

Levels in Reverse Engineering

The reverse engineering process takes effect through the following levels:

1. Abstraction level: The design information of the software system or application is derived at the highest level possible.

2. Completeness: At the above abstraction level, the details of the system are obtained.

3. Interactivity: The degree of the human integration with 5the automated reverse engineering tools is measured.

4. Directionality: It can be either:

(a)  One way: All the extracted information is given to the sogftware engineering who is doing the maintenance.

(b)  Or two way: All the extracted information is fed to a re- engineering tool which then regenerates the old software program.

      5. Extract abstraction: From the old source the processing specifications are obtained.

     

Stages in Reverse Engineering Process

      Reverse engineering process from sommerville goes through the following stages:

      1. System to be re- engineered is subjected to automated analysis.

      2. Then it is manually annotated.

      3. With the system information obtained, a whole set of new documentation is generated containing:

       (a)  Program structure diagrams 

       (b)  Data structure diagrams and 

       (c)   Trace ability matrices.

     

Activities in Reverse Engineering Process

     There are 3 basic activities involved with the reverse engineering process:

   

    1. Understanding process: In order to understand the procedural abstractions as well as functionality analyze the source code on the level:

      (a)  System

      (b)  Program 

      (c)   Component 

      (d)  Statement and 

      (e)  Pattern

    

       2. Understanding data:  It analyzes internal data structures and data base structure.  

   3. User interfaces: It analyzes the basic actions processed by the interface, system’s behavioral response to those actions and equivalency of the interfaces.

When is Reverse Engineering Preferred?

- Reverse engineering is usually preferred when the specifications and designs of the system are required to carry out the program maintenance activities.

- The re- engineering process is preceded by the reverse engineering. 

- To put it simply, the reverse engineering can be thought of as a process that goes back through the whole development cycle.

- UML is a source that supports reverse engineering. 

- Different people have their own different perceptions of the reversion engineering. 

- It can also be thought of as an inversion of the water fall model of software development. 

Software Reverse Engineering Techniques

Software reverse engineering is done to retrieve the source code of a program because the source code was lost, to study how the program performs certain operations, to improve the performance of a program, to fix a bug (correct an error in the program when the source code is not available), to identify malicious content in a program such as a virus or to adapt a program written for use with one microprocessor for use with another. Reverse engineering for the purpose of copying or duplicating programs may constitute a copyright violation. In some cases, the licensed use of software specifically prohibits reverse engineering.

- De compilers
These are programs which will convert object code back to high level languages such as C.
- Functional Analysis
The input and output states of a chip can be monitored using an oscilloscope, or special purpose probes such as logic state analyzers or protocol analyzers, to acquire a picture of the behavior of the chip over time or in response to input signals.
- Patents
Many patented goods are not sold with restrictive licenses, and hence a bonafide purchaser cannot usually be prevented by the patent from doing what they like with the patented product. Indeed, the patent itself may give the reverse engineer valuable information on how the patented product operates.
- Software anti-tamper technology
It is used to deter both reverse engineering and re-engineering of proprietary software and software-powered systems. In practice, two main types of reverse engineering emerge. In the first case, source code is already available for the software, but higher-level aspects of the program, perhaps poorly documented or documented but no longer valid, are discovered. In the second case, there is no source code available for the software, and any efforts towards discovering one possible source code for the software are regarded as reverse engineering.
- Analysis through observation of information exchange, most prevalent in protocol reverse engineering, which involves using bus analyzers and packet sniffers, for example, for accessing a computer bus or computer network connection and revealing the traffic data thereon. Bus or network behavior can then be analyzed to produce a stand-alone implementation that mimics that behavior.
- Disassembly using a disassembler, meaning the raw machine language of the program is read and understood in its own terms, only with the aid of machine-language mnemonics.

Hardware Reverse Engineering Techniques

Reverse engineering is taking apart an object to see how it works in order to duplicate or enhance the object. The practice, taken from older industries, is now frequently used on computer hardware and software.
Hardware reverse engineering involves taking apart a device to see how it works. In general, hardware reverse engineering requires a great deal of expertise and is quite expensive.

- REFAB (Reverse Engineering - Feature Based)
This tool uses a laser digitizer to scan the part, and the analysis software then analyzes the shape of the part, using features which are based on typical machining operations, to generate a computerized manufacturing description which can be displayed, used to copy the product, or produce new products using the design.

- PRINTED CIRCUIT BOARDS (PCBS)
Computer vision has been widely used to scan PCBs for quality control and inspection purposes, and based on this, there are a number of machine vision for analysing and reverse engineering PCBs.

- INTEGRATED CIRCUIT (IC) COMPONENTS
* The first step is to get through the encapsulating material into the product itself, by chemical etching or grinding.
* Once at the chip surface, each layer of components is photographed, then ground away to reveal the layer below. This process reveals the structure of the chip.
* Although these processes can reveal the structure of the chip, they do not indicate the voltages at each point. However, if the chip is undamaged, voltage contrast electron microscopy can be used to scan the chip in use, and watch the voltage level change over time.
These processes are generally referred to as "stripping" or "peeling" the chip.

Overview of Reverse Engineering

Reverse engineering is the attempt to recapture the top level specification by analyzing the product - I call it an "attempt", because it is not possible in practice, or even in theory, to recover everything in the original specification purely by studying the product.

Reverse engineering is difficult and time consuming, but it is getting easier all the time thanks to IT, for two reasons:
- Firstly, as engineering techniques themselves become more computerised, more of the design is due to the computer. Thus, recognisable blocks of code, or groups of circuit elements on a substrate, often occur in many different designs produced by the same computer program. These are easier to recognise and interpret than a customised product would be.
- Secondly, artificial intelligence techniques for pattern recognition, and for parsing and interpretation, have advanced to the point where these and other structures within a product can be recognized automatically.

Reverse engineering generally consists of the following stages:
1. Analysis of the product
2. Generation of an intermediate level product description
3. Human analysis of the product description to produce a specification
4. Generation of a new product using the specification.
There is thus a chain of events between the underlying design specification and any intermediate level design documents lying behind the product, through the product itself, through the reverse engineered product description, through the reverse engineered specification, and into the new product itself.

Reasons for reverse engineering:

- Interoperability.
- Lost documentation: Reverse engineering often is done because the documentation of a particular device has been lost (or was never written), and the person who built it is no longer available. Integrated circuits often seem to have been designed on obsolete, proprietary systems, which means that the only way to incorporate the functionality into new technology is to reverse-engineer the existing chip and then re-design it.
- Product analysis : To examine how a product works, what components it consists of, estimate costs, and identify potential patent infringement.
- Digital update/correction : To update the digital version (e.g. CAD model) of an object to match an "as-built" condition.
- Security auditing.
- Military or commercial espionage : Learning about an enemy's or competitor's latest research by stealing or capturing a prototype and dismantling it.
- Removal of copy protection, circumvention of access restrictions.
- Creation of unlicensed/unapproved duplicates.
- Academic/learning purposes.
- Curiosity.
- Competitive technical intelligence.
- Learning.