month we took at look at the "deep web" or the part of the web
that's not usually included in most general purpose search engines.
A couple of companies have recognized this and
offer products that claim to dig deeper than what you might find
using a conventional search engine.
Method to the Madness
I thought I had my testing all mapped out; when I
first started the last article I fired up one of the deep web bots
and searched for a web site I set up for a local non-profit on a
volunteer basis (evergreenarboretum.com
for the curious). This site hadn't been listed in any search engine
so it was a perfect test. The first bot I tested, LexiBot 1.0 from
BrightPlanet worked like a pro and dug up the site.
Since then, no doubt to my incredible influence,
the site has been added to about seven search engines so my original
test wouldn't be valid.
But I pressed on and decided to use a search term
of "Everett" and Arboretum" and see what turned up.
I love Copernic;
I've been using since it first came out and use it almost as often
as I use Google. I fed Copernic the two search terms and got 40
hits. However only about four of them actually applied or contained
both search terms. I used that as my standard.
LexiBot is BrightPlanet's search bot and a free 30
day evaluation copy can be downloaded from
BrightPlanet The purchase price is $89.95. LexiBot claims to
access more than 2,600 "deep" web databases organized into nearly
200 different channels. While I can't independently verify this my
sense is that it is searching a very wide variety of places. More on
Installation is easy and comes with very
impressive documentation. There are a ton of options and the only
one I played seriously with was the fast vs. quality setting which
allows you to trade off speed (the numbers of sites searched) vs.
the potential results.
The display screen is easy to read and displays a
score, rank, title, URL and size field.
LexiBot returned 72 hits and this is where it
really gets neat. If you click on the "status" envelope icon and the
view in browser or view as text button you get a mini snapshot of
the site. Here's a sample of the text view:
Of course you can click on the URL and go to the
actual web site in your browser, but the "quick view" feature is
incredibly handy. I wish that there was a way to scan through all
the site "snapshots" quickly and perhaps there is and I missed it,
but it's a great feature and a huge time saver vs. surfing to the
actual site only to find it's not exactly what you're looking for.
If you click on the "advanced tools" button
(advanced tools are enabled on the screen shot above) you get
additional info and views includes stats, rejects and other info.
Results can be saved and recalled and you can add notes making it a
good "theme favorites" tool.
BullsEye is Intelliseek's entry into the search
bot arena. A 30 day evaluation version is available from
Intelliseek and can be purchased for $49.99.
Right out of the box it reminded me a lot of
On my arboretum and everett search it turned up 6
sites (ignore the 31 matches, don't know where that came from unless
it was a total count of hits). What impressed me most was the
viewer. It was easy to access from each entry with just a single
click. I found this a lot easier than the method used by Lexibot.
What wasn't obvious to me was the places BullsEye
searched. It was my impression that, like Copernic, it only hit a
fixed set of search engines. While it had an impressive set of
features I felt the results were light and despite the added
features I got much better results in Google.
The Bottom Line
Despite my small nit about the way Lexibot showed
"mini" results my sense is that its searches are deeper and more
comprehensive than anything else I've seen. There is a version 2
near release and while I don't have any vision into what it might or
might not do what I saw in 1.0 encourages me that BrightPlanet is on
the right track.
Both products are available for evaluation but I'd
start with Lexibot. It's my choice for the serious search tool box
and once you use it I don't think you'll ever be satisfied by
Until next month best in computing.