|
Sort Order:
Is there any workaround for the memory (and performance!) problem? I tried to create clients with XFire and Axis2, but both of them failed with strange errors when trying to generate the client code. Shouldn't the WebService just work with any WS-Toolkit? note that this problem exists with the jira-soapclient sample code, not with JIRA itself. As a note, the only way we could finally have a working solution is by doing "Page scraping" and by building a HTML POST message containing the attachment. It is not very elegant, nor practical, but this way we were able to upload any file with the same efficiency as with the web interface. Hello Xavier, thanx for the info. Can you tell me how to build an URL for doing so, This is definitely a server-side problem that does affect SOAPpy: >>> soap.addAttachmentsToIssue(auth, "TP-1", ["largefile.txt"], [[file_contents]])
<Fault soapenv:Server.generalException: java.lang.OutOfMemoryError: Java heap space; nested exception is:
java.lang.OutOfMemoryError: Java heap space: <SOAPpy.Types.structType detail at 146593164>: {'hostname': 'psyche', 'faultData': ''}>
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/var/lib/python-support/python2.5/SOAPpy/Client.py", line 421, in __call__
return self.__r_call(*args, **kw)
File "/var/lib/python-support/python2.5/SOAPpy/Client.py", line 443, in __r_call
self.__hd, self.__ma)
File "/var/lib/python-support/python2.5/SOAPpy/Client.py", line 357, in __call
raise p
SOAPpy.Types.faultType: <Fault soapenv:Server.generalException: java.lang.OutOfMemoryError: Java heap space; nested exception is:
java.lang.OutOfMemoryError: Java heap space: <SOAPpy.Types.structType detail at 146593164>: {'hostname': 'psyche', 'faultData': ''}>
>>>
By setting -XX:+HeapDumpOnOutOfMemoryError I can get a memory dump when JIRA OOMs, and it's all Axis classes using the memory Until we get this fixed, a workaround is to use plain HTTP to upload files. Eg: $ curl -s 'http://localhost:8080/secure/AttachFile.jspa?id=10010&os_username=test&os_password=test' -F filename.1=@largefile.txt > /dev/null Hi Ralf, here is the function, in c#, doing the upload with a HTTP POST, its probably not perfect, but it does the job : private bool UploadFileWithPost( string filename, string issueID ) // Create the bytes array for the file to upload. HttpWebRequest webRequestRFC = (HttpWebRequest)WebRequest.Create(loginUrl); // Auth with server, and fetch cookie for session ID // Create 2nd request // Generate random boundary, should really be random rather than a fixed string like this string stringDataRFC = stringDataRFC += "Content-Disposition: form-data; name=\"filename.1\"; filename=\"" + filename + "\"\n" + string strEndTag = "\n" + boundary + "\n"; byte[] dataRFCbytes = Encoding.UTF8.GetBytes(stringDataRFC); webRequestRFC2.Method = "POST"; // Copy file int bufSize = 1024; int read = 0; // Copy end of tag return true; } } It (=Axis encoding of every byte as a node) probably explains an extremely long upload time. In test run a 100K file takes 10 min+ to get attached to TST project on Atlassian site. Oh, my! To send a small file of 3K Axis generates 700K SOAP message! An excerpt (full version of the message see in the attachment): <in3 soapenc:arrayType="xsd:byte[][1]" xsi:type="soapenc:Array" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"> CRAZY!!! P.S. Axis 1.4 This API is essentially unusable for anything except small text files. Got the same problem in Jira 3.8.1. Shouldn't the filecontent be defined as xsd:base64binary instead of byte array in wsdl? For those who need to do it from Java: a quick'n'dirty addAttachmentToIssue method using Apache HttpClient 3.0.1: Work both via proxy and directly. Interesting that I had to disable transfer-encoding, otherwise JIRA corrupts attaches. ===== import java.io.File; import org.apache.commons.httpclient.Credentials; import td2jira.Config; public class AddAttachmentToIssue { public static void addAttachmentToIssue(JIRAIssue jiraIssue,String fileName, byte[] data) { private static void auth(HttpClient httpclient) { HostConfiguration hc = httpclient.getHostConfiguration(); if( Config.HTTP_PROXY_USER != null ) { int backSlash = Config.HTTP_PROXY_USER.indexOf(' private static void doLogin(HttpClient httpclient) throws Exception { int code = httpclient.executeMethod(login); private static void doAttach(HttpClient httpclient,String issueId,String fileName,byte[] data) throws Exception { FileOutputStream fos = new FileOutputStream(tmp); String attachUrl = Config.JIRA_URL+"/secure/AttachFile.jspa?id="+issueId; PostMethod attach = new PostMethod(attachUrl); Part[] parts = { new StringPart("multiple", "false"), new StringPart("id", issueId), new StringPart("Attach", "Attach"), new StringPart("comment", Config.SYNC_ATTACH_COMMENT_PREFIX+fileName), new StringPart("commentLevel", ""), new FilePart("filename.1", tmp) }; // NB: gotta do it, otherwise JIRA corrupts attachments and mangles comments!!! attach.setRequestEntity(new MultipartRequestEntity(parts, attach.getParams())); httpclient.executeMethod(attach); ===== Am I understanding Jeff's comment When did u get the fix for this problem? Please write a date I don't see a fix. I think the FileTransportFormat is already Base64: But perhaps the FileByteArray as a normal invoke parameter is not a good idea! Sorry, I'm not very involved in the SOAP area. Here's the key parts of upload attachments via Post in Ruby: Unable to find source-code formatter for language: ruby. Available languages are: javascript, sql, xhtml, actionscript, none, html, xml, java #!/bin/ruby
require 'httpclient'
require 'mime/types'
require 'cgi'
password= IO.read('./password').chomp
password= CGI::escape(password)
username= "username"
jira = "http://jira.example.com:8080"
id = "22741" # You'll need to fetch this with the SOAP client for an issue key
url = jira + '/secure/AttachFile.jspa?id=' + id + '&os_username=' + username + '&os_password=' + password
# From http://wiki.rubyonrails.org/rails/pages/HowToSendAFileByPostMultipart
# with updates for httpclient2
file1 = File.open('image.gif')
form = { 'filename.1' => file1, 'comment' => "Some Comment" }
boundary = Array::new(8){ "%2.2d" % rand(42) }.join('__')
extheader = {'content-type' => "multipart/form-data; boundary=___#{ boundary }___"}
client = HTTPClient.new
response = client.post_content(url, form, extheader)
Hi Guys, There is a better way to fix this than HTTP request. I have the issue that in our system we use LDAP SSO integration - so Vladimir's fix doesn't work - I can't log on to LDAP SSO reliably with HTTP Request - It keeps telling me my ticket tokens are incorrect etc. So, i've been trying to work out how to fix this for the last few days - and the simplest answer is to write a small fix in the plugin, and to use the method jiraSoapService.addAttachmentsToIssue almost as intended. The Fix is as follows:
The code for the client side is as follows... It assumes that you have a String[] strNames, and a byte[][] bteBinaries ready to go for submitting to the addAttachmentsToIssue method. ---------------------------------- //Workaround for AXIS - Eugene Koutsenko 22/07/08 String[] tmpFiles = new String[strNames.length*2]; for (i=0; i < strNames.length; i++) { tmpFiles[i*2] = strNames[i]; tmpFiles[i*2+1]= org.apache.axis.encoding.Base64.encode(bteBinaries[i]); } strNames = tmpFiles; //End Workaround jiraSoapService.addAttachmentsToIssue(authToken,returnedIssue.getKey(),strNames,bteBinaries); ---------------------------------- And at the server side, you will have to modify IssueServiceImpl.java inside atlassian-jira-rpc-plugin-xxx.jar - it's under services. The mod is in the addAttachmentsToIssue method - basically we have to decode the information sent by the code above, and put it back into the familiar format expected by the RPC plugin - and then let it do its thing. ---------------------------------- //Fix for AXIS issue with passing binaries (http://jira.atlassian.com/browse/JRA-11693 if (fileNames.length % 2 != 0) throw new RemoteValidationException("Number of items in fileNames is not even. Expected: Names in odd members, Base64 byte arrays in even."); String[] tmpFileNames = new String[fileNames.length / 2]; for (int i = 1; i <= fileNames.length / 2; i++) { tmpFileNames[i-1] = fileNames[i*2-2]; tmpAttachments[i-1] = org.apache.axis.encoding.Base64.decode(fileNames[i*2-1]); } fileNames = tmpFileNames; //End Fix //... Continue on with the rest of method... --------------------------------- That's it. Just tested it out locally - works like a charm. Hope it helps! Eugene. Nice patch Eugene Would you mind uploading a patched JAR file for 3.12.3 for those of us who have yet to configure a jira build environment? The C# version of your patch is as follows (untested but it should just work)... private void AddAttachmentsToIssue(string issuekey, string[] filenames) { //Workaround for AXIS - Eugene Koutsenko 22/07/08 string[] tmpfiles = new string[filenames.Length * 2]; byte[] data; for (int i = 0; i < filenames.Length; i++) { tmpfiles[i * 2] = System.IO.Path.GetFileName(filenames[i]); data = File.ReadAllBytes(filenames[i]); tmpfiles[i * 2 + 1] = Convert.ToBase64String(data, Base64FormattingOptions.InsertLineBreaks); } filenames = tmpfiles; //End Workaround _jss.addAttachmentsToIssue(_token, issuekey, filenames, new sbyte[1] { 0 }); } It seems rather memory heavy given its not chunked, but it looks like it should work. As I prefer not to change the semantic of an existing method, I have added a new 'addBase64AttachmentsToIssue' method to the JiraSoapService public boolean addBase64AttachmentsToIssue(String token, String issueKey, String[] fileNames, String[] base64Attachments) throws RemotePermissionException, RemoteAuthenticationException, RemoteException, RemoteValidationException { User user = tokenManager.retrieveUserNoPermissionCheck(token); // Convert Base64 String into byte[] byte[][] attachments = new byte[base64Attachments.length][]; for (int i = 0; i < base64Attachments.length; i++) { attachments[i] = org.apache.axis.encoding.Base64.decode(base64Attachments[i]); } return issueService.addAttachmentsToIssue(user, issueKey, fileNames, attachments); } You will find the patched version based on 3.12.3 as attachment 'atlassian-jira-rpc-plugin-3.12.3-1.jar' To test it, you can use the following Python script: #!/usr/bin/env python import sys, getpass, datetime, base64 from SOAPpy import SOAPProxy # Arguments: http://localhost:8080/rpc/soap/jirasoapservice-v2?wsdl admin print "SOAP test ", sys.argv[1], " user ", sys.argv[2] soap = SOAPProxy(sys.argv[1]) jirauser=sys.argv[2] # Ask for password passwd=getpass.getpass("Password for %s: " % jirauser) auth = soap.login(jirauser, passwd) issueKey = 'TST-1'; fileName = "MyFile.txt"; issue = soap.getIssue(auth, issueKey) print "Retrieved issue:", issue print print "Adding attachment.." f = open(fileName); myattachment = base64.encodestring(f.read()); f.close(); soap.addBase64AttachmentsToIssue(auth, issueKey, [ fileName ], [myattachment] ); sys.exit(); Thank you for this implementation idea to work-around the problem Is this fix included in the atlassian-jira-rpc-plugin-3.13-1.jar version. What is the solution for 13? This fix is not in the atlassian-jira-rpc-plugin-3.13-1.jar. A solution has been incorporated into JIRA 3.13.3. The fix is an additional method on the JiraSoapService interface: /**
* An alternative mechanism for adding attachments to an issue. This method accepts the data of the attachments as
* Base64 encoded strings instead of byte arrays. This is to combat the XML message bloat created by Axis when
* SOAP-ifying byte arrays.
*
* For more information, please see <a href="http://jira.atlassian.com/browse/JRA-11693">JRA-11693</a>.
*
* @see #addAttachmentsToIssue(String, String, String[], byte[][])
* @param token the SOAP authentication token.
* @param issueKey the issue to attach to
* @param fileNames an array of filenames; each element names an attachment to be uploaded
* @param base64EncodedAttachmentData an array of Base 64 encoded Strings; each element contains the data of the
* attachment to be uploaded
* @return true if attachments were successfully added; if the operation was not successful, an exception would be
* thrown
* @throws RemotePermissionException If the user is not permitted to perform this operation in this context.
* @throws RemoteAuthenticationException If the token is invalid or the SOAP session has timed out
* @throws RemoteValidationException If there is a problem with the group name, such as if the group name exists.
*/
boolean addBase64EncodedAttachmentsToIssue(String token, String issueKey, String[] fileNames, String[] base64EncodedAttachmentData)
The old method (addAttachmentsToIssue) has been deprecated. Check out the commits in Fisheye for full details of the fix: http://svn.atlassian.com/fisheye/changelog/public/atlassian/rpc-jira-plugin The new method requires the calling client to do some extra work, that is base64 encode the file data before sending it across. If you fail to base64 encode the data then it will be "mis-decoded" by the server. Be aware of this. The XML footprint and hence the memory usage profile of this method greatly improves file attachment via SOAP. Can I get an atlassian-jira-rpc-plugin-3.13-3.jar without upgrading to 3.13.3 full JIRA. I am currently running 3.12.2 #335 @ Antony Van Alphen The plugin has only really been verified to work on 3.13. For example the following method RemoteProject[] getProjects(String token) throws RemoteException; was removed for major performance reasons. So if you used it then you would not be able to use it. We can only support the RPC plugin running on the platforms it has been tested on. And they pair up to a JIRA version. That said the plugin is open source, so technically you can take the new attachment method from the current plugin and back port it to the old one. I know it uses the same underlying code with a base64 decode step. You can get the file atlassian-jira-rpc-plugin-3.12.3-1.jar attached to that ticket to run with your 3.12.2 release. It should work. I am using an older version of Jira (3.6.2) and was able to get this to work as well after downloading the source. The only diff was that I used User user = tokenManager.retrieveUser(token); public boolean addBase64EncodedAttachmentsToIssue(String token, String issueKey, String[] fileNames, String[] base64Attachments) throws RemotePermissionException, RemoteAuthenticationException, RemoteException, RemoteValidationException { User user = tokenManager.retrieveUser(token); // Convert Base64 String into byte[] byte[][] attachments = new byte[base64Attachments.length][]; for (int i = 0; i < base64Attachments.length; i++) { attachments[i] = org.apache.axis.encoding.Base64.decode(base64Attachments[i]); } return issueService.addAttachmentsToIssue(user, issueKey, fileNames, attachments); } Thanks. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Bug
Resolved
Critical
SOAP: addAttachmentsToIssue runs out of memory when adding attachments of 1.5 MB or larger
(( And the network is constantly busy all this time, which suggests a VERY BIG xml.
The problem appears to be that axis 1.3 in its wisdom uses a jurassic amount of RAM compared to the attachment size. The OOME on the client happens probably because the overhead is too much of storing a DOM containing many elements and attributes to describe every single byte in the attachment.
Axis 2 purports to feature a low memory overhead so this may be the way forward for jira-soapclient.