A couple of companies have recognized this and offer products that claim to dig deeper than what you might find using a conventional search engine.

Method to the Madness

I thought I had my testing all mapped out; when I first started the last article I fired up one of the deep web bots and searched for a web site I set up for a local non-profit on a volunteer basis (evergreenarboretum.com for the curious). This site hadn't been listed in any search engine so it was a perfect test. The first bot I tested, LexiBot 1.0 from BrightPlanet worked like a pro and dug up the site.

Since then, no doubt to my incredible influence, the site has been added to about seven search engines so my original test wouldn't be valid.

But I pressed on and decided to use a search term of "Everett" and Arboretum" and see what turned up.

Copernic

I love Copernic; I've been using since it first came out and use it almost as often as I use Google. I fed Copernic the two search terms and got 40 hits. However only about four of them actually applied or contained both search terms. I used that as my standard.

LexiBot

LexiBot is BrightPlanet's search bot and a free 30 day evaluation copy can be downloaded from BrightPlanet The purchase price is $89.95. LexiBot claims to access more than 2,600 "deep" web databases organized into nearly 200 different channels. While I can't independently verify this my sense is that it is searching a very wide variety of places. More on that later.

Installation is easy and comes with very impressive documentation. There are a ton of options and the only one I played seriously with was the fast vs. quality setting which allows you to trade off speed (the numbers of sites searched) vs. the potential results.

The display screen is easy to read and displays a score, rank, title, URL and size field.

LexiBot returned 72 hits and this is where it really gets neat. If you click on the "status" envelope icon and the view in browser or view as text button you get a mini snapshot of the site. Here's a sample of the text view:

Of course you can click on the URL and go to the actual web site in your browser, but the "quick view" feature is incredibly handy. I wish that there was a way to scan through all the site "snapshots" quickly and perhaps there is and I missed it, but it's a great feature and a huge time saver vs. surfing to the actual site only to find it's not exactly what you're looking for.

If you click on the "advanced tools" button (advanced tools are enabled on the screen shot above) you get additional info and views includes stats, rejects and other info. Results can be saved and recalled and you can add notes making it a good "theme favorites" tool.

BullsEye Pro

BullsEye is Intelliseek's entry into the search bot arena. A 30 day evaluation version is available from Intelliseek and can be purchased for $49.99.

Right out of the box it reminded me a lot of Copernic:

On my arboretum and everett search it turned up 6 sites (ignore the 31 matches, don't know where that came from unless it was a total count of hits). What impressed me most was the viewer. It was easy to access from each entry with just a single click. I found this a lot easier than the method used by Lexibot.

What wasn't obvious to me was the places BullsEye searched. It was my impression that, like Copernic, it only hit a fixed set of search engines. While it had an impressive set of features I felt the results were light and despite the added features I got much better results in Google.

The Bottom Line

Despite my small nit about the way Lexibot showed "mini" results my sense is that its searches are deeper and more comprehensive than anything else I've seen. There is a version 2 near release and while I don't have any vision into what it might or might not do what I saw in 1.0 encourages me that BrightPlanet is on the right track.

Both products are available for evaluation but I'd start with Lexibot. It's my choice for the serious search tool box and once you use it I don't think you'll ever be satisfied by anything else.

Until next month best in computing.

Back to article index