One of the key steps in KDD is to create a suitable target data set for the data mining tasks. In web data mining, data can be collected at several sites, such as proxy servers, web servers, or an organization’s operational data-bases, which contain business data or consolidated web log data. Web data mining has the same objective as data mining in that both attempt to search for valuable and meaningful knowledge from databases or data warehouses.
However, web data mining differ from data mining in that the former is a more unstructured task than the latter. The difference is based on the characteristics of web documents or web log files which represent unstructured relationships with little machine-readable semantics, while data mining is aimed at dealing with a more structured database.
In recent years, several web search engines were suggested as the advent of web technology. Since 1960, those search engines have been credited with many achievements in the field of information retrieval, such as index modeling, document representation and similarity measure. Recently, some researchers applied database concept to the web database and presented some new
methods of modeling and querying web content at a finger granularity level instead of a page level. Nevertheless, web data mining is concerned with discovering patterns or knowledge from web documents or web log files.
As shown in Figure 1, web data mining is classified into roughly three domains: web content mining, web structure mining, and web usage mining.
Pyle (1999) and Srivastava et al. (2000) presented a detailed taxonomy for web usage mining methods and systems. Web content mining is the process of extracting knowledge from the content of a number of web documents. Web content mining is related to using web search engines, the main role of which is to discover web contents according to the user’s requirements and constraints.
In recent years, the web content mining approach of using the traditional search engine has migrated into intelligent agent-based mining and database-driven mining, where intelligent software agents for specific tasks support the search for more relevant web contents by taking domain characteristics and user profiles into consideration more intelligently. They also help users interpret the discovered web contents.
Many agents for web content mining appeared in literature such as Harvest (Brown et al., 1994), FAQ-Finder (Hammond et al., 1995), Information Manifold (Kirk et al., 1995), OCCAM (Kwok & Weld, 1996), and ParaSite (Spertus, 1997). The techniques used to develop agents include various information retrieval techniques (see Frakes & Baeza-Yates, 1992; Liang &
Huang, 2000), filtering and categorizing techniques (see Broder et al., 1997;
Chang & Hsu, 1997; Maarek & Shaul, 1996; Bonchi et al., 2001), and individual preferences learning techniques (see Balabanovic et al., 1995; Park et al., 2001). Database approaches for web content mining have focused on techniques for organizing structured collections of resources and for using standard database querying mechanisms.
Figure 1. Taxonomy of Web Data Mining (Adapted from Pyle, 1999, and Srivastava et al., 2000)
As to the query language, Konopnicki and Shmueli (1995) combined structure queries based on the organization of hypertext documents, and combined content queries based on information retrieval techniques. Lakshmanan et al. (1996) suggest a logic-based query language for restructuring to extract information from web information sources. On the basis of semantic knowl-edge, efficient ways of mining intra-transaction association rules have been proposed by Ananthanarayana et al. (2001) and Jain et al. (1999). A frame metadata model was developed by Fong et al. (2000) to build a database and extract association rules from online transactions stored in the database. Web log data warehousing was built by Bonchi et al. (2001) to perform mining for intelligent web caching.
Web structure mining is the process of inferring knowledge from the organization and links on the Web, while web usage mining is the automatic discovery of user access patterns from web servers. Our approach is belonging to web usage mining because we are aimed at proposing the way of amplifying the inference value from the web log files, which potential users left through surfing the target web site. Web structure includes external structure, internal structure, and URL itself. External structure mining is therefore related with investigating hyperlinked relationships between web pages under consider-ation, while internal structure mining analyzes the relationships of information within the web page. URL mining is to extract URLs that are relevant to decision maker’s purpose. Spertus (1997) and Chakrabarti et al. (1999) proposed some heuristic rules by investigating the internal structure and the URL of web pages. Craven et al. (1998) used first-order learning technique in categorizing hyperlinks to estimate the relationship between web pages. Brin and Page (1998) considered citation counting of referee pages to find pages that are relevant on particular topics. To mine the community structure on the Web, Kumar et al. (1999) proposed a new hyperlink analysis method. Zaiane (2001) presented building virtual web views by warehousing the web structure that would allow efficient information retrieval and knowledge discovery.
Web usage mining applies the concept of data mining to the web log file data, and automatically discovers user access patterns for a specific web page.
Web usage mining can also use referrer logs as a source. Referrer logs contain information about the referring pages for each page reference, and user registration or survey data gathered via CGI scripts (Jicheng et al., 1999). The results of web usage mining give decision makers crucial information about the life time value of customers, cross-marketing strategies across products, and the effectiveness of promotional campaigns. Among other things, web usage mining helps organizations analyze user access patterns to targeted ads or web
pages, categorize user preferences, and restructure a web site to create a more effective management of workgroup communication and organizational infra-structure.
Web usage mining provides the core basis for our system by supporting customized web usage tracking analysis and psychographics analysis. This customized web usage tracking analysis focuses on optimizing the structure of web sites based on the co-occurrence patterns of web pages (Perkowitz &
Etzioni, 1999), predicting future HTTP request to adjust network and proxy caching (Schechter et al., 1998), deriving marketing intelligence (see Buchner
& Mulvenna, 1999; Cooley et al., 1997, 1999; Spiliopoulou & Faulstich, 1999; Hui & Jha, 2000; Song et al., 2001), and predicting future user behavior on a specific web site by clustering user sessions (see Shahabi et al., 1997; Yan et al., 1996; Changchien & Lu, 2001; Lee et al., 2001). Psychographics analysis, which gives insights about the behavioral patterns of specific web site visitors, requires data about routes taken by visitors through a web site, the time spent on each page, route differences based on differing entry points to the web site, the aggregated route behavior, and general click stream behavior, etc.
(Cooley et al., 1997, 1999). Based on these data, the psychographics analysis tries to answer marketing intelligence-related questions about which menu shoppers are using to buy a product, how long shoppers stay in the product description menu before making a decision to buy, and how shoppers feel about specific ads on the Web, etc.
METHODOLOGY
Our proposed hybrid recommendation mechanism is composed of four phases, as shown in Figure 2. The first phase is to extract association rules from the web log database. Among the data mining techniques, association rules mining algorithm has been popular in marketing intelligence fields (Lee et al., 2002). Therefore, we applied association rules mining to the web data mining tasks. The web log database, which has been used in data mining, includes the web surfing log files (time, frequency, duration, products, etc.) users made on a target shopping mall or web site. From a data preprocessing viewpoint, the web log data poses the following challenges: (1) large errors, (2) unequal sampling, and (3) missing values. To remove these noises included in data, we applied preprocessing techniques to web log data. Through web data mining, we can usually find the hidden informative relationships between those products and the interrelated hyperlinks users visited while web surfing. Association
rules are similar to IF-THEN rules, in which a condition clause (IF) triggers a conclusion clause (THEN). In addition, association rules include the support and confidence (Agrawal et al., 1993a, 1993b). The association rules mining algorithm is shown in Table 1.
In the second phase, after the extraction of the association rules, we adapt CBR to extend the quality of reasoning and recover the limitation of rule-based reasoning. CBR is both a paradigm for computer-based problem-solvers and a model of human cognition. Therefore, cases extracted from the customer database may imply the customer’s knowledge of products and predict his future behavior. Through this phase, CBR shows significant promise for improving the effectiveness of complex and unstructured decision-making.
The third phase is to build a hybrid knowledge base. In this phase, we combine rule base with case base. The key features to combining these two different knowledge bases are the customer’s profile and the products.
The final phase of the proposed hybrid recommendation mechanism is to apply inference procedures to the hybrid knowledge base and extract the inference results. Figure 2 shows our proposed mechanism.
IMPLEMENTATION
To prove the quality of the hybrid recommendation mechanism, we implemented the prototype system using the Excel and VBA languages in a Windows XP environment. We call this prototype system CAR (CBR &
Association rule-based Recommendation systems). CAR is composed of five components (Figure 3). The five components are: (1) rule generator, (2) knowledge base, (3) inference engine, (4) justifier, and (5) user interface.
Table 1. Pseudo Code of the Association Rules Mining Algorithm Ck : Candidate transaction set of size k
Lk : Frequency transaction set of size k Lj = {frequent items};
For (k=1; Lk !=∅; k++) Do Begin Ck+1 = Candidates generated from Lk;
For Each transaction t in database Do
Increment the count of all candidates in Ck+1
that are contained in tLk+1 = candidates in Ck+1 with min_support End Return Lk;