Data Mining is an analytic process designed to explore data (usually large amounts of data - typically business or market related) in search of consistent patterns and/or systematic relationships between variables, and then to validate the findings by applying the detected patterns to new subsets of data. The process of data mining consists of three stages:
1. Initial exploration :
This stage usually starts with data preparation which may involve cleaning data, data transformations, selecting subsets of records and - in case of data sets with large numbers of variables ("fields") - performing some preliminary feature selection operations to bring the number of variables to a manageable range. Then, depending on the nature of the analytic problem, this first stage of the process of data mining may involve anywhere between a simple choice of straightforward predictors for a regression model, to elaborate exploratory analyzes using a wide variety of graphical and statistical methods.
2. Model building and validation :
This stage involves considering various models and choosing the best one based on their predictive performance. There are a variety of techniques developed to achieve that goal - many of which are based on so-called "competitive evaluation of models," that is, applying different models to the same data set and then comparing their performance to choose the best. These techniques - which are often considered the core of predictive data mining - include: Bagging (Voting, Averaging), Boosting, Stacking (Stacked Generalizations), and Meta-Learning.
3. Deployment :
The final stage involves using the model selected as best in the previous stage and applying it to new data in order to generate predictions or estimates of the expected outcome.
GOALS OF DATA MINING :
- Prediction : Data mining can show how certain attributes within the data will behave in the future. In such applications, business logic is used coupled with data mining.
- Identification : Data patterns can be used to identify the existence of an item, an event, or an activity. For example, intruders trying to break a system may be identified by the programs executed, files accessed, and CPU time per session.
- Classification : Data mining can partition the data so that different classes or categories can be identified based on combination of parameters.
- Optimization : One eventual goal of data mining may be to optimize the use of limited resources such as time, space, money, or materials and to maximize output variables such as sales or profits under a given set of constraints.
Wednesday, August 12, 2009
Stages and Goals of Data Mining
Posted by
Sunflower
at
8/12/2009 11:15:00 PM
0
comments
Labels: Classification, Data Mining, Deployment, Exploration, Goals of data mining, Identification, Model building, optimization, Prediction
|
| Subscribe by Email |
|
Tuesday, August 11, 2009
Introduction to Data Mining
Data mining is the process of discovering meaningful new correlations, patterns and trends by sifting through large amounts of data stored in repositories, using pattern recognition technologies as well as statistical and mathematical techniques.
Data mining is the analysis of (often large) observational data sets to find
unsuspected relationships and to summarize the data in novel ways that are
both understandable and useful to the data owner.
Data mining is an interdisciplinary field bringing together techniques from
machine learning, pattern recognition, statistics, databases, and visualization to
address the issue of information extraction from large data bases.
WHY MINE DATA ?
1. Commercial View :
- Lots of data is being collected and warehoused.
* Web data, e-commerce.
* Purchases at department/grocery stores.
* Bank/Credit Card transactions.
- Computers have become cheaper and more powerful
* Competitive Pressure is strong.
* Provide better, customized services for an edge.
2. Scientific View :
- Data collected and stored at enormous speeds(GB/hour).
* Remote sensors on a satellite.
* Telescopes scanning the skies.
* Micro arrays generating gene expression data.
* Scientific simulations generating terabytes of data.
- Traditional techniques infeasible for raw data.
- Data mining may help scientists :
* in classifying and segmenting data.
* in Hypothesis Formation.
SCOPE OF DATA MINING :
Data mining derives its name from the similarities between searching for valuable business information in a large database — for example, finding linked products in gigabytes of store scanner data — and mining a mountain for a vein of valuable ore. Both processes require either sifting through an immense amount of material, or intelligently probing it to find exactly where the value resides. Given databases of sufficient size and quality, data mining technology can generate new business opportunities by providing these capabilities:
- Automated prediction of trends and behaviors.
- Automated discovery of previously unknown patterns.
Automated discovery of previously unknown patterns.
* More columns : Analysts must often limit the number of variables they examine when doing hands-on analysis due to time constraints. Yet variables that are discarded because they seem unimportant may carry information about unknown patterns. High performance data mining allows users to explore the full depth of a database, without preselect a subset of variables.
* More rows : Larger samples yield lower estimation errors and variance, and allow users to make inferences about small but important segments of a population.
Posted by
Sunflower
at
8/11/2009 11:43:00 PM
0
comments
Labels: Data, Data Mining, introduction, Scope, Why data mining
|
| Subscribe by Email |
|
Monday, August 10, 2009
Functionality of Data Warehouses and Building a Data Warehouse
Data warehouses exist to facilitate complex, data-intensive, and frequent adhoc queries. The data warehouse access component supports enhanced spreadsheet functionality, efficient query processing, structured and adhoc queries, data mining, and materialized views. These offer pre-programmed functionalities such as :
- Roll-up : Data is summarized with increasing generalization.
- Drill-down : Increasing levels of details are revealed.
- Pivot : Cross tabulation is performed.
- Slice and dice : Performing projection operations on the dimensions.
- Sorting : Data is sorted by ordinal value.
- Selection : Data is available by value or range.
- Derived attributes : Attributes are computed by operations on stored and derived values.
BUILDING A DATA WAREHOUSE :
In constructing a data warehouse, builders should take a broad view of the anticipated use of the data warehouse. Acquisition of data for the warehouse involves the following steps :
- The data must be extracted from different sources.
- Data must be formatted for consistency within the data warehouse. Names, meanings, and domains of data from unrelated sources must be reconciled.
- Data must be cleaned to ensure validity.
- The data must be fitted into the data model of the data warehouse.
- The data must be loaded into the warehouse.
- How up-to-date must be data be ?
* Can the warehouse go off-line, and for how long.
* What are the data interdependencies ?
* What is the storage availability ?
* What are the distribution requirements ?
* What is the loading time ?
Data warehouses must also be designed with full consideration of the environment in which they will reside. Important design considerations include the following :
- Usage projections.
- The fit of the data model.
- Characteristics of available sources.
- Design of the metadata component.
- Modular component design.
- Design for manageability and change.
- Consideration of distributed and parallel architecture.
Posted by
Sunflower
at
8/10/2009 10:28:00 PM
0
comments
Labels: Building a data warehouse, Data Model, data warehousing, Functionality
|
| Subscribe by Email |
|
Sunday, August 9, 2009
Introduction To Data Warehousing
A data warehouse is a type of computer database that is responsible for collecting and storing the information of a particular organization. The goal of using a data warehouse is to have an efficient way of managing information and analyzing data.
In addition to a relational database, a data warehouse environment includes an extraction, transportation, transformation, and loading (ETL) solution, an online analytical processing (OLAP) engine, client analysis tools, and other applications that manage the process of gathering data and delivering it to business users.
Despite the fact that data warehouses can be designed in a number of different ways, they all share a number of important concepts.
A data warehouse is a :
- Subject-oriented : This means that the information that is in the data warehouse is stored in a way that allows it to be connected to objects or events which occur in reality.
- Integrated : Data warehouses must put data from disparate sources into a consistent format. They must resolve such problems as naming conflicts and inconsistencies among units of measure. When they achieve this, they are said to be integrated.
- Time-variant : A time variant will allow changes in the information to be monitored and recorded over time.
- Non-volatile : This means that it cannot be deleted, and must be held to be analyzed in the future. All of the programs that are used by a particular institution will be stored in the data warehouse, and it will be integrated together. 
CHARACTERISTICS OF DATA WAREHOUSES :
- Multi-dimensional conceptual view.
- Generic dimensionality.
- Unlimited dimensions and aggression levels.
- Unrestricted cross-dimensional operations.
- Dynamic sparse matrix handling.
- Client-server architecture.
- Multi-user support.
- Accessibility.
- Transparency.
- Intuitive data manipulation.
- Consistent reporting performance.
- Flexible reporting.
Posted by
Sunflower
at
8/09/2009 11:53:00 PM
0
comments
Labels: Characteristics, Concepts, data warehousing, Process
|
| Subscribe by Email |
|
Relationships And Relationship Sets
A relationship is an association between several entities.
A relationship set is a set of relationships of the same type.
A relationship type R among n entity types E1, E2,....., En defines a set of associations or a relationship set among entities from these types.
In ER diagrams, relationship types are displayed as diamond-shaped boxes, which are connected by straight lines to the rectangular boxes representing the participating entity types. The relationship name is displayed in the diamond-shaped box.
RELATIONSHIP DEGREE : The degree of a relationship type is the number of participating entity types. A relationship type of degree two is called binary, and one with degree three is called ternary.

ROLE NAMES and RECURSIVE RELATIONSHIPS :
Role names signifies the role that a participating entity from the entity type plays in each relationship instance, and helps to explain what the relationship means.
Role names are not necessary in relationship types where all the participating entity types are distinct, since each entity type name can be used as a role name. However, in some cases, the same entity type participates more than once in a relationship type in different roles. In such cases the role name becomes essential for distinguishing the meaning of each participation. Such relationships are called recursive relationships.
MAPPING CONSTRAINTS :
An E-R scheme may define certain constraints to which the contents of a database must conform.
* Mapping Cardinality : It expresses the number of entities to which another entity can be associated via a relationship. For binary relationship sets between entity sets A and B, the mapping cardinality must be one of:
1. One-to-one: An entity in A is associated with at most one entity in B, and an entity in B is associated with at most one entity in A.
2. One-to-many: An entity in A is associated with any number in B. An entity in B is associated with at most one entity in A.
3. Many-to-one: An entity in A is associated with at most one entity in B. An entity in B is associated with any number in A.
4. Many-to-many: Entities in A and B are associated with any number from each other.
The appropriate mapping cardinality for a particular relationship set depends on the real world being modeled.
* Existence Dependencies: if the existence of entity X depends on the existence of entity Y, then X is said to be existence dependent on Y. (Or we say that Y is the dominant entity and X is the subordinate entity.)
For example,
o Consider account and transaction entity sets, and a relationship log between them.
o This is one-to-many from account to transaction.
o If an account entity is deleted, its associated transaction entities must also be deleted.
o Thus account is dominant and transaction is subordinate.
Posted by
Sunflower
at
8/09/2009 10:49:00 PM
0
comments
Labels: Binary degree, Constraints, Databases, Degree, ER model, Recursive, Relationship sets, Relationships, Ternary degree
|
| Subscribe by Email |
|
Thursday, August 6, 2009
Entity - Relationship Model
The entity-relationship (ER) data model allows us to describe the data involved in a real-world enterprise in terms of objects and their relationships and is widely used to develop an initial database design. The ER model is important primarily for its role in database design. It provides useful concepts that allow us to move from an informal description of what users want from their database to a more detailed, and precise, description that can be implemented in a DBMS.
ENTITIES, ATTRIBUTES, AND ENTITY SETS
- A database can be modeled as:
o a collection of entities,
o relationship among entities.
- An entity is an object that exists and is distinguishable from other objects.
o Example: specific person, company, event, plant
- Entities have attributes.
o Example: people have names and addresses
- An entity set is a set of entities of the same type that share the same properties.
o Example: set of all persons, companies, trees, holidays.
- Attributes : An entity is represented by a set of attributes, that is descriptive properties possessed by all members of an entity set.
o E.g. Employee = (Name, Address, Age, Salary)
Types of Attributes
- SIMPLE attributes are attributes that are drawn from the atomic value domains.
E.g. Name = {John} ; Age = {23}
- COMPOSITE attributes: Attributes that consist of a hierarchy of attributes.
E.g. Address may consists of “Number”, “Street” and “Suburb” → Address = {59 + ‘Meek Street’ + ‘Kingsford’}
- SINGLE VALUED attributes: Attributes that have only one value for each entity.
E.g. Name, Age for EMPLOYEE
- MULTIVALUED attributes: Attributes that have a set of values for each entity.
E.g. Degrees of a person: ‘ BSc’ , ‘MIT’, ‘PhD’
- DERIVED attributes: Attributes Contain values that are calculated from other attributes.
E.g. Age can be derived from attribute DateOfBirth. In this situation, DateOfBirth might be called Stored Attribute.
![]() | ![]() | ![]() |
- DOMAIN/VALUE SETS : Each simple attribute of an entity type is associated with a value set which specifies the set of values that may be assigned to that attribute for each individual entity.
- KEYS : An important constraint on the entities is the key. An entity type usually has an attribute whose values are distinct for each individual entity in the collection. Such an attribute is called a key attribute.
For example, for the entity SET EMPLOYEE = {EID, Name, Address, Age, Salary}
EID attribute is the unique key for entity set EMPLOYEE as no two employees can have two same EID. Some entity types have more than one key attribute. An entity type may also have no key.
Posted by
Sunflower
at
8/06/2009 12:41:00 PM
0
comments
Labels: Attributes, Database design, Domain, Entity, Entity Set, Key, Types
|
| Subscribe by Email |
|
Conceptual Data Models for Database Design
Conceptual modeling is an important phase in designing a successful database application. The database design process consists of a number of steps listed below:
Step 1: Requirements Collection and Analysis
- Prospective users are interviewed to understand and document data requirements
This step results in a concise set of user requirements, which should be detailed and complete.
- The functional requirements should be specified, as well as the data requirements. Functional requirements consist of user operations that will be applied to the database, including retrievals and updates.
- Functional requirements can be documented using diagrams such as sequence diagrams, data flow diagrams, scenarios, etc.
Step 2: Conceptual Design / Data Modeling
- Once the requirements are collected and analyzed, the designers go about creating the conceptual schema.
- Conceptual schema: concise description of data requirements of the users, and includes a detailed description of the entity types, relationships and constraints.
- The concepts do not include implementation details; therefore the end users easily understand them, and they can be used as a communication tool.
- The conceptual schema is used to ensure all user requirements are met, and they do not conflict.
Step 3: Database Design
- Two sub-steps called Database Logical Design which define a database in a data model of a specific DBMS and Database Physical Design which define the internal database storage structure are defined.
- It also defines file organization or indexing techniques.
Step 4: Database Implementation
- Many DBMS systems use an implementation data model, so the conceptual schema is transformed from the high-level data model into the implementation data model.
- This step is called logical design or data model mapping, which results in the implementation data model of the DBMS.
Step 5: Physical Design
- Internal storage structures, indexes, access paths and file organizations are specified.
- Application programs are designed and implemented.
Posted by
Sunflower
at
8/06/2009 12:09:00 PM
0
comments
Labels: Conceptual data model, conceptual design, Database design, database implementation, Databases, Design, physical design, Requirement analysis, Steps
|
| Subscribe by Email |
|


