This is a critical issue for Japanese users - I would say the highest priority issue among open issues related to Japanese internationalization. Can we target it for a fix in 2.6.1 if possible?

Thanks,

Neeraj

This is classed as an improvement, so it doesn't qualify for fixing in a stable release like 2.6.1. I believe the problem with searching for single-byte characters is related to Lucene's internal encoding, so it's not a trivial fix either.

I'll make sure your comment is flagged with our product manager.

Regards,
Matt

Appreciate your guidance. I rated this as blocker from sales and market development perspective since we cannot be the best enterprise wiki in the class if our search simply does not find matching documents. For Japanese customers, this is the same as if search does not work. I am fine as long as this gets addressed within a reasonable time-frame because we are getting a lot of heat from our customers on this issue.

cheers, Neeraj

Thanks for providing the example documents. I am afraid I am not sure if understand the desired behavior completely.
Is the following example in line with your expectations?

Steps:

1) Create a page containing the full width character カ.
2) Perform a search for カ.
3) 1 result should be returned. (This appears to work correctly)
4) Search for the string ｶ. (half width character)
5) No results are returned (but you require / expect the same result obtained in step 3).

Please advise me if this behavior you are expecting. Unfortunately I have little experience with the Japanese language so I may be misinterpreting your request.

Thanks,
Andrew

Thanks for looking into this. Let me explain as below:

PART 1 (Critical, applies to half-width katakana characters):
1) Create a page containing the half width character ｶ.
2) Perform a search for ｶ.
3) 1 result should be returned. (zero results are being returned erroneously)

PART 2 (High, applies to both half-width and full-width katakana characters):
4) Now create a new page containing the full width character カ
5) Perform a search for カ.
6) 2 results should be returned. (currently only 1 result (full-width) is returned)
7) Search for the string ｶ. (half width character)
8) 2 results should be returned. (currently zero results are returned)

PART 3 (High, applies to both half-width and full-width numeric characters):
9) Now create a page containing the full width character １and another page containing the half-width character 1.
10) Perform a search for full-width１.
11) 2 results should be returned. (currently only 1 result (full-width) is returned)
12) Search for the half-width 1.
13) 2 results should be returned. (currently only 1 result (full-width) is returned, as in step 11)

Please let me know if you have further questions.

regards,
Neeraj

I am unable to produce the behavior you have indicated for part 1. I have attached a screen shot of correct results being returned for a page containing a half width katakana character "ka".
That being said, I have discovered this result will only be returned if the top search bar is used (i.e. not the search in the search tab), so we may well have a bug which is leading to the results you have experienced.
Which search bar are you using to perform the search?
In addition, could you provide us with the system information from the administration tab of the machine you are encountering this problem on?

For part 3, when entering characters in both full width and half width, I was unable to retrieve any results. How exactly were you entering these characters?

Thanks,
Andrew

Attached is the system information of the machine the client tested this problem on. I have not verified this for all cases, and assume the client was searching via the search tab.

Regarding the half and full-width numeric characters , you may well be right.

I guess now atleast you are as aware of the problem as I am. We basically need to ensure that no matter which search box is used, Confluence is able to search for half and full width katakana and numeric characters correctly – i.e it should pass tests for part 1-3.

Let me know how I can help further.

Neeraj

I realized that I was using the English indexer when performing my searches. After switching to CJK indexing, I was able to produce results consistent with your findings. I will investigate further.

Lucene's CJKAnalyzer is definitely not indexing half width characters correctly. I have raised an issue (http://issues.apache.org/jira/browse/LUCENE-1032) to address this.
We were considering creating a patch ourselves, but the simplest implementation would require usage of Java 6's Normalizer class. In order to solve this, I have created our Analyzer, Custom Japanese Analyzer. Unfortunately this only works on Sun JDKs and so it will not be incorporated into Lucene and may not work for all customers.

Customers who are experiencing problems such as the ones you outlined should use this Analyzer in place of CJKAnalyzer until the issue with Lucene is resolved, assuming they have a Sun JDK.

1. Where can I get the Custom Japanese Analyzer and what are the installation instructions for it?

2. Regarding the Lucene support issue you mention above, I did not see any mention of the problems with half-width Japanese characters.

Please clarify.

regards,

Neeraj

My apologies, I provide the wrong issue number. It should be LUCENE-1032.

The custom Japanese analyzer will be available as an option in the Content Indexing tab from 2.6.2 onwards.

Regards,

Andrew Lynch

Also, to confirm, will search for half width katakana and numeric characters work properly from the top search bar as well as from the search tab?

The changes were not too numerous, so I should be able to provide a patch for 2.6.0 if essential.

The search tab has not yet been fixed, I have created another issue (CONF-9834) which should be watched for updates.

Thanks.