This lawsuit cuts to the heart of fair use and the copyright control of databases. The comments on Threadwatch appear divided - particularly with regards to whether opt-out (via robots.txt) is sufficient.
However for most search engine features, opt-out is clearly sufficient: Fair use protects your right to quote from and refer to a work by title and other distinguishing features, or to describe facts about a work or from a work in general. The typical search engine listing is well within the limit, and if the search engines wanted to, they could likely safely ignore robots.txt with no legal consequences.
The features that are more interesting are Google's cache, which is an outright copy, and their aggregation of data that a company such as Agence France Presse may try to assert a database copyright for, since they provide a syndication service.
This is more likely to involve database copyrights.
In the US, a database can only be protected if it shows originality in it's selection, coordination and arrangement. This means that a purely automated aggregation of all news items provided by partners for instance likely would not receive any protection.
However even then, extracting facts from a database is legal, short of duplicating the structure of the whole database.
Under EU law, databases have sui generis protection, that is, you can't reuse data from a database while it is protected (protection lasts for 15 years), even if the data is pure facts. If you want to use these facts you must compile them yourself from other sources. If Google have used AFP's aggregated data to compile the news, this may well be illegal under French law - however it seems weird for AFP to sue in California, as the US Supreme Court have explicitly rejected the idea of such protection under current US copyright law.
The paradox is that Google (and anyone else) clearly have a legal right to publish the location, descriptions and titles of these news articles under fair use, but that there is a chance that aggregating them may violate AFP's copyrights depending on how it's done, and what jurisdiction we're talking about.
(Btw. I am not a lawyer - don't rely on me to decide whether or not it's safe to aggregate data...)