Full Text Search of mulitple language using Neutral Word Breaker

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

Soh Chee King - 09 May 2008 02:13 GMT

Hi,

To enable support of FTS of multiple languages, we are using neutral word
breaker in ms sql 2000 for our application.

Unfortunately, there seems to be a limitation in the word breaker. It does
not support part content of words which is crucial for chinese search.

Is there a fix or alternative solution for searching both chinese and
english characters?

Reply to this Message

Hilary Cotter - 09 May 2008 12:31 GMT

No, they should be stored in different columns and use different word
breakers for each column.

> Hi,
>
[quoted text clipped - 7 lines]
> Is there a fix or alternative solution for searching both chinese and
> english characters?

Reply to this Message

Soh Chee King - 12 May 2008 03:39 GMT

Hi Hilary,

If a column contains both chinese and english characters (E.g. I will not
be able to split the information into one column chinese and one column
english), I will not be able to search across the two languages?

Even if I store in different columns, it will still be the same content
(With english and chinese characters) and the search will not be correct.

Please advise.

Thanks.

> No, they should be stored in different columns and use different word
> breakers for each column.
[quoted text clipped - 9 lines]
> > Is there a fix or alternative solution for searching both chinese and
> > english characters?

Reply to this Message

Hilary Cotter - 12 May 2008 11:54 GMT

You will be able to search on both languages, but the results will be
unpredicatable. If you use the Chinese word breaker on your index, the
English words might not be stored correctly in the index. If you use the
English word breaker the Chinese characters will not be stored correctly in
the index.

Then if you search in Chinese you will not find what you are looking for if
you index in English - English searches will work. If you search in Chinese
and index in Chinese Chinese searches will work and English FreeText
searches will not.

> Hi Hilary,
>
[quoted text clipped - 23 lines]
>> > Is there a fix or alternative solution for searching both chinese and
>> > english characters?

Reply to this Message