newbie: Full Text Search Against PDF Blobs

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

Des - 14 Aug 2007 01:12 GMT

I have a client which this solution sounds perfect for:

Does anyone know a web site that I can test the "Full Text Search
Against PDF Blobs" functionality against.

The website "layout design guy" is saying that SQL 2005 will be too slow
and we should use "Lucene", an open source indexer instead.
Does anyone have any info I can use to show this guy that SQL Server
2005 will be faster?

The target site has several thousand Report PDFs at 2Mb average each
(about 10GB in total).

I have watched the video
"http://download.microsoft.com/download/b/3/8/b3847275-2bea-440a-8e2e-305b009bb26
1/sql_13.wmv"
that was referenced in this group recently.

Thanks,
Des

Reply to this Message

Hilary Cotter - 14 Aug 2007 03:58 GMT

With Lucene you really have to roll your own solution, all it is, is a
full-text search engine. You have to write code to query it and to feed
documents to it to index these documents. Lucene is designed for the 5-10
million document range, but can be scaled much higher. It is optimized to
return results in batches to 10, 20, 25 or 100 results. If you return all
results its performance is much worse than SQL FTS.

Lucene allows you to so true property based searches.

SQL FTS is highly scalable but you really have to think about partitioning
after you hit 50 million rows.

You really have to test to see what works best in your environment.

Signature

relevantNoise - dedicated to mining blogs for business intelligence.

Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html

Looking for a FAQ on Indexing Services/SQL FTS
http://www.indexserverfaq.com

>I have a client which this solution sounds perfect for:
>
[quoted text clipped - 15 lines]
> Thanks,
> Des

Reply to this Message

Des - 14 Aug 2007 07:07 GMT

Thanks Hillary,

That was very useful indeed.
I would much prefer to stay inside the environments I am familiar with,
if possible.

Do you know of a site that uses SQL FTS to Search PDF's Stored as Blobs?
It would be nice to know that any "known" shortcomings, before I attempt
to implement it, otherwise I will be thinking it was something I did.

Thanks,
Des

> With Lucene you really have to roll your own solution, all it is, is a
> full-text search engine. You have to write code to query it and to feed
[quoted text clipped - 9 lines]
>
> You really have to test to see what works best in your environment.

Reply to this Message

Hilary Cotter - 14 Aug 2007 11:56 GMT

No, but I know a site that use sql fts to index over 2 terrabytes.

Signature

> Thanks Hillary,
>
[quoted text clipped - 22 lines]
>>
>> You really have to test to see what works best in your environment.

Reply to this Message

Simon Sabin - 18 Aug 2007 11:41 GMT

Hello Des,

There was a shortcoming that there was no 64 bit iFilter that has now been
resolved http://sqlblogcasts.com/blogs/simons/archive/2007/07/18/PDF-64-bit-iFilter-at-la
st.aspx

We index a few million CVs in wor format no problem, I wouldn't see any difference
with PDFs

Simon Sabin
SQL Server MVP
http://sqlblogcasts.com/blogs/simons

> Thanks Hillary,
>
[quoted text clipped - 23 lines]
>>
>> You really have to test to see what works best in your environment.

Reply to this Message

Des - 18 Aug 2007 16:43 GMT

Thanks Simon - Thats good to know :-)

Regards
Des

> Hello Des,
>
[quoted text clipped - 8 lines]
> SQL Server MVP
> http://sqlblogcasts.com/blogs/simons

Reply to this Message