Return to Articles

Column 19, September 1996

Copyright 1996 Robert J. Ambrogi

`Crawling' For Needles In The Internet Haystack

By Robert J. Ambrogi

There is a lot of information out there in Cyberspace -- if only you can find it.

Organization of the Internet is anarchic, to say the least. The resource you need could as easily be on a major university's Web site as on some high school kid's home page.

Not so long ago, the only way to find resources on the Internet was using Archie or Veronica -- not the cartoon characters, but software that allowed you to search particular sites for specific file names.

Now, there are tools that allow you to search through virtually the full text of everything on the World Wide Web for specific words and concepts.

These tools use what are called "crawlers" or "spiders." Think of crawlers as little Web-surfing robots, gathering everything they find into an enormous database. That database becomes a searchable index of almost everything available on the Web.

What follows is a guide to these search tools. The list is not exhaustive, but offers an introduction to crawlers, both general and legal.


Among the best of the crawlers are these:

The `Ambrogi' Test

It's not scientific, but it works for me. Given a somewhat uncommon surname, I use it as my search-engine acid test -- the more matches for "ambrogi," the more thorough the database. In a recent search, here were the results:


Think of these as one-stop shopping. They allow you to use several search engines from one location, and sometimes in one search.

Legal Searchers

The problem with crawlers is that they can return too much information. A key-word search of the Web can produce thousands of matches. Law crawlers index only law-related Web sites, making your search more targeted and the matches more likely to be relevant to your query.

Other Finding Tools

Robert J. Ambrogi, a lawyer in Rockport, Mass., is editor of, a monthly newsletter about the Internet published by Legal Communications Ltd., Philadelphia. He can be reached by e-mail at or by phone at (978) 546-7898.