Focused Web Crawler and its Approaches

Abstract

There has been a rapid growth of the world-wide web which has scaled beyond our imaginations. To surmount these challenges search engines are used. One of the most important type of crawler is Focused crawler which is used to index information according to a particular topic. To maximize the possibility of downloading relevant documents focused crawler makes a prediction of hyperlinks visiting priority which in turn helps to reduce downloading of irrelevant documents and drastically saves network resources and hardware. Instead of using keywords topics are specified by using commendable documents. One of the most important feature of this type of web crawler is collecting and indexing all accessible web credentials. This crawler mainly diagnosis its crawl boundary to search different URLs. In this paper we’ll illustrate a clear cut comparison between focused and standard web crawlers as well as various approaches of focused crawling like contextual and priority based crawling.