Data Mining

For data mining, the data exists within a single table or view where each case (record) is a row.

If you want to mine data stored in a star schema, you still need to create a unique table with a record by case and the variable of interests.

The data mining development process may require several data sets.

A data set may be:

• needed for building (training) the model;
• used for scoring.
• used for testing.

Data Cleansing

The data must be properly cleansed to eliminate inconsistencies and support the needs of the mining application.

Data Discretization

Put data into bin: binning (discretization)

Data Correction

Data Correction - There is always a bad input

Data Normalization

Normalize the data to be able to have:

• a common identifier.
• a common scale

Example:

• Email and URL should be all lowercase
• Metrics should be expressed in a rate rather than in a number

Outlier suppression

outlier suppression is required to not skew the result.