History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: JRA-6679
Type: Bug Bug
Status: Resolved Resolved
Resolution: Cannot Reproduce
Priority: Major Major
Assignee: Dylan Etkin [Atlassian]
Reporter: Nick Menere [Atlassian]
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
JIRA

Large text fields with Sybase being truncated at 16K

Created: 15/May/05 08:00 PM   Updated: 12/Mar/08 07:14 AM
Component/s: Backend / Domain Model
Affects Version/s: 3.1.1
Fix Version/s: 3.4

Time Tracking:
Not Specified

Environment: Sybase DB

Participants: Dylan Etkin [Atlassian], Edel and Nick Menere [Atlassian]
Since last comment: 85 weeks, 2 days ago
Resolution Date: 23/Oct/05 09:43 PM
Labels:


 Description  « Hide
Sue wrote:
> Nick,
>
> Our mappings are identical to yours below.
>
> we are not storing unicode, I just created the workflows with the JIRA administration function in the workflows screen. I think datalength() returns a byte count.
>
> the dba said we cant increase the size of a text field, and we tested the textsize that is being used by JIRA when data is inserted into that table. the textsize is 2147483647 so its not the textsize. are you using INSERT or WRITETEXT to get the data in?
>
> Do we know if anyone else has had this problem with long workflows on a sybase or other database?
>
> Also interestingly the long workflow will work to completion on created issues and seem fine, then when a new workflow is added or one deleted the error will occur and the data is truncated.
>
> Regards,
> Sue.
>
> ----Original Message----
> From: Nick Menere nick.menere@atlassian.com
> Cc: support@atlassian.com
> Subject: Re: Corrupted custom workflows
>
>
> Sue,
> To our knowledge we do not change the data size of any of these rows and
> we don't believe the libraries we use do either.
> By any chance are you storing unicode characters? These take up 2 bytes
> instead of 1. Hence 16K characters would take up 32K bytes. Though I
> am not sure whether datalength() returns a byte count or a character count.
>
> Just to give you a bit background as to how we set up our data types.
>
> In the file - entitymodel.xml we define our mappings from java objects
> to tables. In here you will find a mapping for workflows.
> It looks like this:
> <entity entity-name="Workflow" table-name="jiraworkflows"
> package-name="">
> <field name="id" type="numeric"/>
>
> <field name="name" col-name="workflowname" type="long-varchar"/>
> <field name="creator" col-name="creatorname" type="long-varchar"/>
> <field name="descriptor" type="extremely-long"/>
> <field name="islocked" type="short-varchar"/>
>
> <prim-key field="id"/>
> </entity>
>
> As you can see the descriptor column is mapped to be of type
> "extremely-long"
>
> This value is then mapped to a value in the file fieldtype-sybase.xml
> This value should be:
> <field-type-def type="extremely-long" sql-type="TEXT"
> java-type="String"></field-type-def>
>
> Can you check that yours are mapped correctly.
> Is it possible for you dba to increase the size of these columns - e.g.
> make them 64K?
> I have asked around my colleagues and google and can't find anything on
> our side that would cause this.
>
> Cheers,
> Nick
>
>
>
>
> Sue wrote:
>
>>Nick,
>>
>>This is what our DBA said about my problem. Any idea where JIRA is setting this textsize?.... Sue.
>>
>>the default maximum TEXT size in Sybase is 32K (select @@textsize).
>>
>>you can modify this by using (for example, in bytes).....
>>
>> set textsize 1000000
>>
>>..... before inserting data into the TEXT column.
>>
>>it looks like your program is already modifying it to be a max of 16Kb... eg:
>>
>>1> select datalength(DESCRIPTOR) from jiraworkflows
>>2> go
>>
>> -----------
>> 12813
>> 12813
>> 16384
>> 16384
>> 16384
>> 16384
>> 16384
>> 16384
>>
>>(8 rows affected)
>>1>
>>
>>you just need to find where in the code this can (or rather does) occur, and then make it much larger.
>>
>>
>>Shane
>>
>>----Original Message----
>>From: Nick Menere nick.menere@atlassian.com
>>Cc: support@atlassian.com
>>Subject: Re: Corrupted custom workflows
>>
>>
>>Sue,
>>this seems most weird. The text field in Sybase should be able to
>>handle strings of up to 2 Gig. As big as your workflow is, I doubt it
>>is that big!
>>Are you able to determine the length of the strings that have been
>>truncated?
>>Can you confirm that field can store massive strings? Maybe manually
>>inserting a string into this table then checking it and then removing it?
>>
>>If the database is the issue, one way to fix this would be to store the
>>workflows on disk. Details on how to do this are located at:
>>http://confluence.atlassian.com/display/JIRA/Store+Workflow+on+Disk
>>
>>Cheers,
>>Nick
>>
>>
>>
>>
>>Sue wrote:
>>
>>
>>>Nick,
>>>
>>>Thanks for getting back to me. It is a TEXT type and I found the descriptor column values, it is truncating the long workflows. Does this mean I can only create workflows with under 10 or so steps? These ones had 15 and would often have more.
>>>
>>>Regards,
>>>Sue.
>>>
>>>----Original Message----
>>>From: Nick Menere nick.menere@atlassian.com
>>>Subject: Re: Corrupted custom workflows
>>>
>>>
>>>Hi Sue,
>>>It looks as though the descriptor (stored as xml) may have been cut short.
>>>
>>>There should be entries in the jiraworkflow table in the descriptor
>>>column. Are there no entries for any workflow here?
>>>Was this a large workflow? Many steps?
>>>One thing you may want to check is the size of this column. Is it of
>>>type TEXT and does it allow extremely long strings?
>>>
>>>
>>>Chers,
>>>Nick
>>>
>>>Sue wrote:
>>>
>>>
>>>
>>>>Hi,
>>>>
>>>>I am working on the JIRA Enterprise Edition that we have just bought at ...
>>>>
>>>>I created a custom workflow and a day or so later it disappeared. This >>is the error in the logs when I try to use anything ie. workflow scheme >>that is associated with the workflow.
>>>>
>>>>ole0.bmp
>>>>
>>>>And here is how it appears in the workflow screen, blank with no name.
>>>>
>>>>ole1.bmp
>>>>
>>>>I searched your forums and cannot find anyone with this problem, plus I >>cannot find where the workflow steps are stored in the database or in an >>xml file on the server anywhere. We are using our own Sybase db. I
>>>>found the names of the workflows in jiraworkflows but that was all.
>>>>
>>>>Regards,
>>>>Sue.

 All   Comments   Work Log   Change History      Sort Order: Ascending order - Click to sort in descending order
Nick Menere [Atlassian] - 15/May/05 08:05 PM
No-one has reported this issue before. Have been looking around on google and can find a few references to Sybase having a 16K issues for certain drivers but no official documentation.

http://blogs.law.harvard.edu/philg/comments?u=philg&p=7706&link=http://blogs.law.harvard.edu%2Fphilg%2F2005%2F03%2F07%23a7706#a7710
http://www.object-craft.com.au/pipermail/python-sybase/2005-April/000392.html

This has been an issue with Oracle databases as before they released 10g they had a well known 4K text limit. Though there is workarounds for that.

Options for workaround are:
1) there is mention that the Sybase JDBC can do this.

2) Store your workflows on disk:
http://confluence.atlassian.com/display/JIRA/Store+Workflow+on+Disk


Dylan Etkin [Atlassian] - 23/Oct/05 09:43 PM
Tested an issue with a 64k description on Sybase 12.5.1. I used the jconn2.jar and jTDS2.jar jars that came from sybase. There was no problem with the field being truncated, every thing seemed to work fine.

Nick Menere [Atlassian] - 29/Nov/06 05:16 PM
We now recommend customers use the jtds driver as the Sybase driver seemed to be the cause of the problem.

Edel - 29/Nov/06 08:28 PM
Nick - I spoke to you this morning in realation to this Issue which we are still encountering. I have spoken again to our DBA and he is under the opinion that it is not drive realated but controled by the client - this is what he says, can you please Review and let em know your thoughts.

*******************************************************************************************************************************************
Yep - most people miss the point with Sybase and the TEXT datatype.

Sybase can store up to 2Gb per TEXT column. There is no concept of changing the database itself to allow varying amounts of text data to be stored. This is something each client application needs to control.

For example, if you use the Sybase supplied tools (isql, Sybase Central, etc), the default limit these tools impose is 32K... ie:

1> select @@textsize
2> go

-----------
32768

(1 row affected)
1>

If you use DBArtisan (a 3rd party query tool) the limit is as follows:

select @@textsize
10485760

The point being that each application controls how much data can be stored and/or retrieved in a TEXT column.

To change this, you simply issue the "set textsize" command from the client application - an example being:

set textsize 2000000


Nick Menere [Atlassian] - 29/Nov/06 11:28 PM
Hi,

From the documentation on "textsize", this apparently sets the maximum amount of data the client will receive back. In this case the client is the JDBC driver. Presumably the jTDS driver sets this to something sensibly high automatically, whereas the Sybase driver doesn't.

Google doesn't find much about setting textsize. One page [1] suggests that it can be done by appending:

";SQLINITSTRING=set TextSize 32000"

(without quotes) to the JDBC URL.

If you get it working, please let us know. Otherwise, using jTDS is the simplest option.

[1] http://www.sybase.com/detail?id=47902