KDD-Cup 2000: Question 1
Winner’s Report
Amdocs
Aron Inger, Nurit Vatnik, Saharon Rosset, Einat Neumann
{aroni, nuritv, saharonr, einatn} @Amdocs.com
1.	THE KDD-CUP QUESTION
The training database contained 234,954 sessions, some complete and some clipped. We were asked to build a prediction model to describe which sessions are going to continue and which are about to terminate, and use this model to predict the outcomes on a test dataset (where the information on whether the sessions were clipped or not was withheld). The prediction problem was particularly difficult for several reasons, most notably:
1.	The skewed class distribution (over 80% of the sessions were complete) together with the scarcity of powerful predictive attributes meant it was very difficult to improve over the “majority” model.
2.	The test data was from a different period, so it was difficult to tell which of the patterns discovered in the training data would persist. 
It should be noted, however, that both these issues are very relevant when dealing with real applications and handling them correctly may be an important ingredient in success.
2.	METHODOLOGY AND ALGORITHMS
Our analysis approach is based on the combination of automated and manual analysis, integrating the power of the machine with expert knowledge and human insight. In this case, we can divide the analysis and modeling into alternating stages of manual and automated knowledge discovery. The main tools we used were the proprietary Amdocs Business Insight (BI) tool and SAS (mainly for data manipulation and simple reporting). 
As with every analysis, the data must be firstly organized and understood (the data was generally clean so very little auditing was required). In this step we examined the data using SAS and the basic visualizations of the BI tool. The first major observation we made was that 60% of the data were one-click sessions. These were special since the one-click full sessions were mostly bots and crawlers, which always have a full session length of only one click, and are very easy to identify (they announced themselves). The remaining one-click sessions must still be identified as either clipped or not clipped. We made a decision at this point to separate the one-click session from all the rest, a decision that had proven itself to be very useful later. In this first stage we also investigated the need to generate additional transformations to those supplied in the aggregated data files. Except for a few gaps we used the aggregated data as given.
The next stage was to run the automated knowledge discovery algorithm of the BI tool on the data, and analyze its results. At Amdocs we place great emphasis on the algorithms being “white box” and giving interpretable, understandable results. The user can try to understand and interpret the rules by changing them, adding/deleting conditions and creating new rules based on the insights generated from these rules. This also helps the user identify biases and problems in the data. Analyzing the knowledge discovery results gave us a few key insights about the data and about the prediction model we should be building. 
For the one-click sessions, where 87% of the sessions were complete, there were almost no patterns describing sessions that are about to continue. With the exception of five small segments (see Figure 1) for which we were able to predict continuance, we concluded that all other one-click sessions should be automatically classified as complete. 
Most of the multi-click sessions (65%) were complete and produced many more useful patterns. We were able to identify an interesting set of patterns from hints in the output of the automated algorithms. Sessions whose referring site was “file://c:…” kept appearing in small but significant patterns. We identified these patterns as originating from internal testing scripts. Since there were a small number of scripts, with many sessions generated by each, we were able to identify with almost perfect certainty for each of these sessions whether it was complete or clipped.  In this way about 1% of the data could be marked as ‘continue’ with near perfect accuracy.  The discovery of these patterns illustrates the power of human-machine synergy.
3.	THE FINAL MODEL 
The final model was built by combining automatically generated prediction models with user knowledge acquired through the entire process, as described in Figure 1

Figure 1 Final Prediction Model
For the one-click sessions, it involved separating the bots and then using a manually crafted set of rules on the remainder of the sessions, for predicting “continue” (e.g.: all identified customers who had personal details). For the multi-click data we manually modeled the testing patterns discussed above, and on the remaining data, built an ensemble prediction model based on three different models. The three models all use a decision tree as their basis. “Best Rule” model chooses the most accurate rule satisfied by each record (and gives the expected accuracy of this rule as a score). “Merged” model performs a logistic regression algorithm on the rule-set, and defines the score as a weighted combination of the rules a record satisfies. “Hybrid” model is similar to the “Merged” model, but also uses raw field values in the logistic regression. The three scores are averaged to create a more robust final score.