Introduction to Machine Learning (part-5)

Data Mining : It defined as a process used to extract usable data from a larger set of any raw data. It implies analyzing data patterns in large batches of data using one or more software.

Crisp-dm  known for Cross-Industry Standard process for Data Mining. Aim is to develop a tool and application neutral process for conducting data mining and define tasks, outputs from these tasks, terminology and mining problem type characterization. It has four level of abstraction :

  • Phases
          • Example: Data Preparation 

  • Generic Tasks 
          • A Stable, general and complete set of tasks                                                              • Example: Data Cleaning 

  • Specialized Task 
         • How is the Generic task carried out                                                                           • Example: Missing Value Handling 

  • Process Instance 
        • Example: The mean value for numeric attributes and the median for                        categorical attributes was used.

Data Mining Context : In data mining the context is defined by four dimensions
  • Application domain: Medical Prognosis.
  • Data Mining Problem Type: Regression. 
  • Technical Aspect: Censored Observations.
  • Tools and Techniques: Cox’s Regression, CIL’s GENNA.
The context of the data mining task at hand is the starting point for mapping the generic tasks to specific tasks required in this instance. When actual event of interest is not observed, it is known as Censored observation.

Phases of Crisp-dm :




It is non - linear, there is repeatedly backtracking.

Business Understanding Phase:
  • Understand the business objectives It need to understand the business objective, the business process ,cost/benefit analysis. 
  • Current Systems Assessment : Identify the key actors, minimum the Sponsor and the Key User, What forms should the output take? , Integration of output with existing technology landscape and Understand Market norms and standards.
  • Task Decomposition : Break down the objective into sub-tasks and Map sub-tasks to data mining problem definitions
  • Identify Constraints : Resources and Law e.g. Data Protection
  • Build a project plan :List assumptions and risk (technical/financial/business/ organisational) factors.
Data Understanding Phase :
  • Collect Data : Internal and External Sources (e.g. Axiom, Experian), Document reasons for inclusion/exclusions, Depend on a domain expert and Accessibility issues.
  • Data Description : Document data quality issues, requirements for data preparation and Compute basic statistics.
  • Data Exploration : Simple univariate data plots/distributions,Investigate attribute interactions and Data Quality Issues.
The Data Preparation Phase :
  • Integrate data: Joining multiple data tables and Summarisation/aggregation of data.
  • Select Data : Attribute subset selection : Rationale for Inclusion/Exclusion and Data sampling : Training/Validation and Test sets.
  • Data Transformation : Using functions such as log , Factor/Principal Components analysis and Normalization/Discretisation.
  • Clean Data : Handling missing values/Outliers.
  • Data Construction : Derived Attributes.
The Modelling Phase:

  • Select of the appropriate modelling technique: Data pre-processing implications on attribute independence and Data types/ Normalisation/ Distributions.It is  dependent on  Data mining problem type and Output requirements.
  • Develop a testing regime : By sampling,Verify samples have similar characteristics and are representative of the population.
  • Build Model : Choose initial parameter settings and Study model behavior.
  • Assess the model : Beware of Over-fitting ,Investigate the error distribution , Identify segments of the state space where the model is less effective , Iteratively adjust parameter settings and then Document reasons of these changes.
The Evaluation Phase :
  • Validate Model : Human evaluation of results by domain experts, Evaluate usefulness of results from business perspective, Define control groups, Calculate Lift curves and Expected Return on Investment.
  • Review Process
  • Determine next steps: Potential for deployment, Deployment architecture and Metrics for success of deployment.
The Deployment Phase :

  • Knowledge Deployment is specific to objectives : Knowledge Presentation , Deployment within Scoring Engines and Integration with the current IT infrastructure - Automated pre-processing of live data feeds and XML interfaces to 3rd party tools , Generation of a report - Online/Offline and Monitoring and evaluation of effectiveness.
  • Process deployment/productionisation
  • Produce final project Report : Document everything along the way.



Comments

Post a Comment

Popular posts from this blog

Supervised Learning(Part-5)

Image
  Decision Trees (Continued) Attribute Selection Measures If the dataset consists of N attributes then deciding which attribute to place at the root or at different levels of the tree as internal nodes is a complicated step. By just randomly selecting any node to be the root can’t solve the issue. If we follow a random approach, it may give us bad results with low accuracy. For solving this attribute selection problem, researchers worked and devised some solutions. They suggested using some criteria like : Entropy, Information gain, Gini index, Gain Ratio, Reduction in Variance Chi-Square These criteria will calculate values for every attribute. The values are sorted, and attributes are placed in the tree by following the order i.e, the attribute with a high value(in case of information gain) is placed at the root. While using Information Gain as a criterion, we assume attributes to be categorical, and for the Gini index, attributes are assumed to be continuous. Entropy Entropy is a me
Read more

Supervised Learning(Part-2)

Image
  Support Vector Machine A Support Vector Machine (SVM) is a discriminative classifier formally defined by a separating hyperplane. In other words, given labeled training data, the algorithm outputs an optimal hyperplane which categorizes new examples.  In two dimensional space this hyperplane is a line dividing a plane in two parts where in each class lay in either side. Suppose given plot of two label classes on graph as shown in image. You might have come up with something similar to following image (image B). It fairly separates the two classes. Any point that is left of line falls into black circle class and on right falls into blue square   In real world application, finding perfect class for millions of training data set takes lot of time. This is called regularization parameter.  Two terms regularization parameter and gamma are tuning parameters in SVM classifier. Varying those we can achieve considerable non linear classification line with more accuracy in reasonable amount of
Read more

Text Analysis (Part - 4)

Image
  The article covers a very important concepts of Text Analysis from Bayesian's i.e the  Latent Dirichlet Allocation  which is the part of topic modelling . What is Topic Modelling? Topic modelling is a type of statistical modelling for discovering the abstract “topics” that occur in a collection of documents. This can be useful for search engines, customer service automation, and any other instance where knowing the topics of documents is important . Thus, Topic Models , in a nutshell, are a type of statistical language models used for uncovering hidden structure in a collection of texts. There are multiple methods of for topic modelling and one of them includes Latent Dirichlet Allocation (LDA) . Latent Dirichlet Allocation Latent Dirichlet Allocation (LDA) is an example of topic model and is used to classify text in a document to a particular topic. It builds a topic per document model and words per topic model, modelled as Dirichlet distributions. Basic Idea LDA is a  genera
Read more