|
Is there any workaround for the memory (and performance!) problem? I tried to create clients with XFire and Axis2, but both of them failed with strange errors when trying to generate the client code. Shouldn't the WebService just work with any WS-Toolkit?
note that this problem exists with the jira-soapclient sample code, not with JIRA itself.
As a note, the only way we could finally have a working solution is by doing "Page scraping" and by building a HTML POST message containing the attachment. It is not very elegant, nor practical, but this way we were able to upload any file with the same efficiency as with the web interface.
Hello Xavier,
thanx for the info. Can you tell me how to build an URL for doing so, This is definitely a server-side problem that does affect SOAPpy:
>>> soap.addAttachmentsToIssue(auth, "TP-1", ["largefile.txt"], [[file_contents]])
<Fault soapenv:Server.generalException: java.lang.OutOfMemoryError: Java heap space; nested exception is:
java.lang.OutOfMemoryError: Java heap space: <SOAPpy.Types.structType detail at 146593164>: {'hostname': 'psyche', 'faultData': ''}>
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/var/lib/python-support/python2.5/SOAPpy/Client.py", line 421, in __call__
return self.__r_call(*args, **kw)
File "/var/lib/python-support/python2.5/SOAPpy/Client.py", line 443, in __r_call
self.__hd, self.__ma)
File "/var/lib/python-support/python2.5/SOAPpy/Client.py", line 357, in __call
raise p
SOAPpy.Types.faultType: <Fault soapenv:Server.generalException: java.lang.OutOfMemoryError: Java heap space; nested exception is:
java.lang.OutOfMemoryError: Java heap space: <SOAPpy.Types.structType detail at 146593164>: {'hostname': 'psyche', 'faultData': ''}>
>>>
By setting -XX:+HeapDumpOnOutOfMemoryError I can get a memory dump when JIRA OOMs, and it's all Axis classes using the memory Until we get this fixed, a workaround is to use plain HTTP to upload files. Eg:
$ curl -s 'http://localhost:8080/secure/AttachFile.jspa?id=10010&os_username=test&os_password=test' -F filename.1=@largefile.txt > /dev/null Hi Ralf, here is the function, in c#, doing the upload with a HTTP POST, its probably not perfect, but it does the job :
private bool UploadFileWithPost( string filename, string issueID ) // Create the bytes array for the file to upload. HttpWebRequest webRequestRFC = (HttpWebRequest)WebRequest.Create(loginUrl); // Auth with server, and fetch cookie for session ID // Create 2nd request // Generate random boundary, should really be random rather than a fixed string like this string stringDataRFC = stringDataRFC += "Content-Disposition: form-data; name=\"filename.1\"; filename=\"" + filename + "\"\n" + string strEndTag = "\n" + boundary + "\n"; byte[] dataRFCbytes = Encoding.UTF8.GetBytes(stringDataRFC); webRequestRFC2.Method = "POST"; // Copy file int bufSize = 1024; int read = 0; // Copy end of tag return true; } } It (=Axis encoding of every byte as a node) probably explains an extremely long upload time. In test run a 100K file takes 10 min+ to get attached to TST project on Atlassian site.
 (( And the network is constantly busy all this time, which suggests a VERY BIG xml. Oh, my! To send a small file of 3K Axis generates 700K SOAP message!
An excerpt (full version of the message see in the attachment): <in3 soapenc:arrayType="xsd:byte[][1]" xsi:type="soapenc:Array" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"> CRAZY!!! P.S. Axis 1.4
This API is essentially unusable for anything except small text files. Got the same problem in Jira 3.8.1.
Shouldn't the filecontent be defined as xsd:base64binary instead of byte array in wsdl? For those who need to do it from Java: a quick'n'dirty addAttachmentToIssue method using Apache HttpClient 3.0.1:
Work both via proxy and directly. Interesting that I had to disable transfer-encoding, otherwise JIRA corrupts attaches. ===== import java.io.File; import org.apache.commons.httpclient.Credentials; import td2jira.Config; public class AddAttachmentToIssue { public static void addAttachmentToIssue(JIRAIssue jiraIssue,String fileName, byte[] data) { private static void auth(HttpClient httpclient) { HostConfiguration hc = httpclient.getHostConfiguration(); if( Config.HTTP_PROXY_USER != null ) { int backSlash = Config.HTTP_PROXY_USER.indexOf(' private static void doLogin(HttpClient httpclient) throws Exception { int code = httpclient.executeMethod(login); private static void doAttach(HttpClient httpclient,String issueId,String fileName,byte[] data) throws Exception { FileOutputStream fos = new FileOutputStream(tmp); String attachUrl = Config.JIRA_URL+"/secure/AttachFile.jspa?id="+issueId; PostMethod attach = new PostMethod(attachUrl); Part[] parts = { new StringPart("multiple", "false"), new StringPart("id", issueId), new StringPart("Attach", "Attach"), new StringPart("comment", Config.SYNC_ATTACH_COMMENT_PREFIX+fileName), new StringPart("commentLevel", ""), new FilePart("filename.1", tmp) }; // NB: gotta do it, otherwise JIRA corrupts attachments and mangles comments!!! attach.setRequestEntity(new MultipartRequestEntity(parts, attach.getParams())); httpclient.executeMethod(attach); ===== When did u get the fix for this problem? Please write a date
I think the FileTransportFormat is already Base64:
<wsdl:message name="addAttachmentsToIssueRequest"> <wsdl:part name="in0" type="xsd:string"/> <wsdl:part name="in1" type="xsd:string"/> <wsdl:part name="in2" type="impl:ArrayOf_xsd_string"/> <wsdl:part name="in3" type="impl:ArrayOf_xsd_base64Binary"/> </wsdl:message> But perhaps the FileByteArray as a normal invoke parameter is not a good idea! Sorry, I'm not very involved in the SOAP area. Here's the key parts of upload attachments via Post in Ruby:
Unable to find source-code formatter for language: ruby. Available languages are: javascript, sql, xhtml, actionscript, none, html, xml, java #!/bin/ruby
require 'httpclient'
require 'mime/types'
require 'cgi'
password= IO.read('./password').chomp
password= CGI::escape(password)
username= "username"
jira = "http://jira.example.com:8080"
id = "22741" # You'll need to fetch this with the SOAP client for an issue key
url = jira + '/secure/AttachFile.jspa?id=' + id + '&os_username=' + username + '&os_password=' + password
# From http://wiki.rubyonrails.org/rails/pages/HowToSendAFileByPostMultipart
# with updates for httpclient2
file1 = File.open('image.gif')
form = { 'filename.1' => file1, 'comment' => "Some Comment" }
boundary = Array::new(8){ "%2.2d" % rand(42) }.join('__')
extheader = {'content-type' => "multipart/form-data; boundary=___#{ boundary }___"}
client = HTTPClient.new
response = client.post_content(url, form, extheader)
Hi Guys,
There is a better way to fix this than HTTP request. I have the issue that in our system we use LDAP SSO integration - so Vladimir's fix doesn't work - I can't log on to LDAP SSO reliably with HTTP Request - It keeps telling me my ticket tokens are incorrect etc. So, i've been trying to work out how to fix this for the last few days - and the simplest answer is to write a small fix in the plugin, and to use the method jiraSoapService.addAttachmentsToIssue almost as intended. The Fix is as follows:
The code for the client side is as follows... It assumes that you have a String[] strNames, and a byte[][] bteBinaries ready to go for submitting to the addAttachmentsToIssue method. ---------------------------------- //Workaround for AXIS - Eugene Koutsenko 22/07/08 String[] tmpFiles = new String[strNames.length*2]; for (i=0; i < strNames.length; i++) { tmpFiles[i*2] = strNames[i]; tmpFiles[i*2+1]= org.apache.axis.encoding.Base64.encode(bteBinaries[i]); } strNames = tmpFiles; //End Workaround jiraSoapService.addAttachmentsToIssue(authToken,returnedIssue.getKey(),strNames,bteBinaries); ---------------------------------- And at the server side, you will have to modify IssueServiceImpl.java inside atlassian-jira-rpc-plugin-xxx.jar - it's under services. The mod is in the addAttachmentsToIssue method - basically we have to decode the information sent by the code above, and put it back into the familiar format expected by the RPC plugin - and then let it do its thing. ---------------------------------- //Fix for AXIS issue with passing binaries (http://jira.atlassian.com/browse/JRA-11693 if (fileNames.length % 2 != 0) throw new RemoteValidationException("Number of items in fileNames is not even. Expected: Names in odd members, Base64 byte arrays in even."); String[] tmpFileNames = new String[fileNames.length / 2]; for (int i = 1; i <= fileNames.length / 2; i++) { tmpFileNames[i-1] = fileNames[i*2-2]; tmpAttachments[i-1] = org.apache.axis.encoding.Base64.decode(fileNames[i*2-1]); } fileNames = tmpFileNames; //End Fix //... Continue on with the rest of method... --------------------------------- That's it. Just tested it out locally - works like a charm. Hope it helps! Eugene. Nice patch Eugene
Would you mind uploading a patched JAR file for 3.12.3 for those of us who have yet to configure a jira build environment? The C# version of your patch is as follows (untested but it should just work)... private void AddAttachmentsToIssue(string issuekey, string[] filenames) { //Workaround for AXIS - Eugene Koutsenko 22/07/08 string[] tmpfiles = new string[filenames.Length * 2]; byte[] data; for (int i = 0; i < filenames.Length; i++) { tmpfiles[i * 2] = System.IO.Path.GetFileName(filenames[i]); data = File.ReadAllBytes(filenames[i]); tmpfiles[i * 2 + 1] = Convert.ToBase64String(data, Base64FormattingOptions.InsertLineBreaks); } filenames = tmpfiles; //End Workaround _jss.addAttachmentsToIssue(_token, issuekey, filenames, new sbyte[1] { 0 }); } It seems rather memory heavy given its not chunked, but it looks like it should work. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Bug
Verified
Critical
SOAP: addAttachmentsToIssue runs out of memory when adding attachments of 1.5 MB or larger
Axis 2 purports to feature a low memory overhead so this may be the way forward for jira-soapclient.