About
SearchResultFinder is a Firefox plugin to automatically determine XPaths to extract items from a search result page. Given a single search result page (i.e. the search result page you are browsing), the plugin suggests a number of candidate XPaths which you can visually inspect. The XPath can be used to extract items from result pages with the same template.
Visit a search result page.
Right-click
and run.
After running the plugin: a popup appears with suggested XPaths.
If you use this plugin for your scientific work, please cite the following paper:
D. Trieschnigg, K. Tjin-Kam-Jet, and D. Hiemstra. SearchResultFinder: Federated search made easy. In Proceedings of the 36th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2013. July 28–August 1, 2013, Dublin, Ireland, 2013.
Installation
Download the XPI file and open it using Firefox. After restarting Firefox, the popup shown when right-clicking a page should show the option "Run Search Result Finder".
Automating the plugin
Sometimes you don't want to manually open each search result page and invoke the plugin manually. You can use Selenium to automatically open Firefox, open a search result page, run the plugin and retrieve the results.
Examples
You can find a list of example webpages here.Caveats
- When you use the XPath in your programming environment, make sure that the parsed DOM-tree is the same as in Firefox.
For instance, Firefox inserts a
tbodytag into the DOM-tree when handling an HTML table, even when the code does not contain an explicttbodytag. As a result, the plugin may come up with an XPath containingtbodyyielding no results for HTML parsers which do not add atbodyto the DOM-tree. - Note that some search engines use AJAX to load/format the search results. In your programming environment make sure that these AJAX calls are also properly handled.