Page-Level Webpage Menu Detection

Abstract

One of the key elements of a website are Web menus, which provide fundamental information about the structure of the own website. For humans, identifying the main menu of a website is a relatively easy task. However, for computer tools identifying the menu is not trivial at all and, in fact, it is still a challenging unsolved problem. From the point of view of crawlers and indexers, menu detection is a valuable technique, because processing the menu allows these tools to immediately find out the structure of the website. Identifying the menu is also essential for website mapping tasks. With the information of the menu, it is possible to build a sitemap that includes the main pages without having to follow all the links. In this work, we propose a novel method for automatic Web menu detection that works at the level of DOM. Our implementation and experiments demonstrate the usefulness of the technique.