Issue Details (XML | Word | Printable)

Key: JRA-7772
Type: New Feature New Feature
Status: Open Open
Priority: Major Major
Assignee: Unassigned
Reporter: Kevin Wilson
Votes: 34
Watchers: 17
Operations

If you were logged in you would be able to see more operations.
JIRA

Ability to create advanced queries to search across all data

Created: 26/Aug/05 02:13 PM   Updated: Yesterday 10:52 PM
Component/s: Filtering & Indexing, Web interface
Affects Version/s: 3.3
Fix Version/s: None

Time Tracking:
Not Specified

Issue Links:
Part
 
Reference

Participants: AAgave Maketime, Anton Mazkovoi [Atlassian], Christian Foisy, Darrell Wells, Greg Hutton, Jeff Turner [Atlassian], Kevin Wilson, Morgan Filipsson, Nick Menere [Atlassian], Sandor Palfy and Scott Farquhar [Atlassian]
Since last comment: 21 weeks ago
Labels:


 Description  « Hide
Considering the existing search interface does not allow you to query every area of data (e.g., worklog, change history, issue links, etc.) and some areas (custom fields) cannot be queried without knowing some preexisting details, it would be very handy to have a search module that allows users to specify their SQL Syntax of choice to be ran against the database. A simple selectable field list that (on select) allows it to have its where clause value(s) specified for it.

 All   Comments   Work Log   Change History      Sort Order: Ascending order - Click to sort in descending order
Scott Farquhar [Atlassian] added a comment - 27/Aug/05 01:24 AM
Kevin,

I'm not sure how this would work, given security limitations that I'm sure you would want to impose on your users. It would also open ourselves to SQL injection attacks.

Perhaps instead of suggesting the solution - you can outline the specific problems that you have? That will make it easier for us to develop a solution.

Cheer,
Scott


Kevin Wilson added a comment - 29/Aug/05 11:57 AM
The security limitations would need to kick in after the search is ran but before the results are displayed. The search interface would need a toggle to be enabled/disabled much like file attachments are. This would allow public installations (like yours) to hide this option but private installations (like ours) to allow it.

I think you have seen all the issues I have submitted that pertain to searching and the problems therein. Either Lucene just doesn't do the job well enough and/or the way it was implemented is flawed. The biggest gripe users have is they cannot find stuff ( Especially Managers). I realize that some of those gripes are user error but there are some problems that I (and others) have reported.

They key problems are searching from scratch with very little knowledge of what you are looking for. Then it moves on to finding a specific issue very fast, users would generally do this using the ID number of the issue. Jira requires the entire key but if you see jra-7777 then you will understand my dilema. Then you move on to returning a specific subset, but 3.3 should provide some relief with the project filtering.


Scott Farquhar [Atlassian] added a comment - 29/Aug/05 06:54 PM
"The biggest gripe users have is they cannot find stuff ( Especially Managers)"

Can you give me examples (aside from the issues that you have already raised) where you would have expected something to be returned from a search and it wasn't?

I'm not sure how giving managers an SQL interface is going to be any easier for them to find stuff, especially when you mention that some of the problems are 'user error'.

Examples of searches that don't work would be very useful.


Kevin Wilson added a comment - 30/Aug/05 09:11 AM
I asked some offenders to send me some recent examples that I can pass on to you. Once I get the feedback I'll create you a login for our site and let you run your tests against our server.

I proposed the sql interface so I can say to them "here, this will let you find anything you could possibly imagine". I am a big proponent of just feeding all the rope I can to the biggest complainers and letting them hang themselves. Then they usually settle down once there is nothing to blame besides themselves.

I can't be too hard on them though since I have found some searching scenarios just aren't possible but I have reported those as you have seen. And if I can't get them to a point where they can accomplish what they want to do then they have the clout to yank jira and replace it with some old proprietary per-seat relic, which they have already hinted at doing.

I have to get my plugins fixed for 3.3.0 so I can deploy the latest jira, just the multi-project select should settle them down a bit.


Kevin Wilson added a comment - 02/Sep/05 12:50 PM
I also found another bug with the quick search...a multiple KEY search in quick search doesn't work unless the key is contained in the Comments of the issue.

Using multiple keys from the same project I only got 1 issue returned. The line below was used, only the 45 is returned because it had its key listed in its comments.

RPWINTWOKXP-51 RPWINTWOKXP-53 RPWINTWOKXP-54 RPWINTWOKXP-55


Jeff Turner [Atlassian] added a comment - 04/Sep/05 09:51 PM
> Technical support passes along the following issue data to Software engineering and labels them as super HOT!
> ...

Technically then, the best solution is to create a custom field called 'Label' and actually label them, so you have something to search on. Would this work in your environment?


Kevin Wilson added a comment - 06/Sep/05 09:06 AM
lol ... the phrase wasn't literal. The bugs already have a status of Critical, adding yet another status and Process step is not going to solve the problem. Allowing multiple keys (or their numbers) to be specified in the quicksearch and somewhere in the search page is all we are looking for here, I fail to see how this is such an unreasonable request since it would bring much joy to all the users of your app.

Nick Menere [Atlassian] added a comment - 06/Sep/05 07:45 PM
Kevin,
We actually think it would be quite a good feature and definietly don't think of it as unreasonable.
The only problem is that we are flat out as it is implementing the issues already scoped for 3.4. We just don't want to give off the impression that this will get done tomorrow as there many other issues out there that the public are screaming for louder than this one.
We try and have a very open procedure when it comes to implementing features:
http://confluence.atlassian.com/display/DEV/Implementation+of+New+Features+and+Improvements

Cheers,
Nick


Jeff Turner [Atlassian] added a comment - 06/Sep/05 11:03 PM
(making title more descriptive)

From the code perspective, what we want is the ability to search for 'key=ABC-123 OR key=ABC-124 OR key=ABC-125 ...'. This falls in the scope of JRA-1560.


Kevin Wilson added a comment - 07/Sep/05 10:16 AM
Guys, just so you know I am a developer too and you usually only hear whining and complaining and never anything good about what you have done to date. Even though I am beating on you guys about this search thing I think you have come leap years since the early days. Jira is a good product and your licensing scheme is what makes it an excellent product. Once you get the management reporting end sorted out then you will have a great product ... just as long as you keep them annual renewal prices under control :-p

Hey Nick,

I realize the squeaky wheel usually gets the grease but there are times where good issues that improve productivity (like issue jra-699) fall through the cracks. That issue wasn't a huge impact at the developer level but on the lead and management level it was critical since they have to keep track of multiple issues in multiple projects, present hot issues in operations meetings and the like. Finding what they need and only what they need in a timely fashion and getting those results into a comprehensive (single) list is imperative. This issue and others I have reported recently build upon that notion. As for those that are screaming loudly I propose an analogy to consider - Slavery in the US would not have been abolished when it was if popular vote was the only criteria to consider.

Jeff,

Actually it's not just for keys in a single project but across multiple projects like ...

key=ABC-123 OR key=ABC-124 OR key=XYZ-124 OR key=QWE-124

See sentences 2 and 3 in my message to Nick.


Anton Mazkovoi [Atlassian] added a comment - 09/Sep/05 02:26 AM
Kevin,

Thanks for the feedback and the kind words! We do our best to ensure that prices stay attractive

Popularity (as it is recorded in JIRA) is definitely not the only factor we look at. The document:
http://confluence.atlassian.com/display/DEV/Implementation+of+New+Features+and+Improvements
tries to make that clear. We are short on resources at the moment, unfortunately, and we are just tryring to ensure that we do not make any promises we cannot keep.

Thanks,
Anton


Kevin Wilson added a comment - 13/Oct/05 10:56 AM
As posted to the Jira mailing list here is a response from the boss ...

----Original Message----
From: Steve E.
Sent: Thursday, October 13, 2005 10:09 AM
To: Kevin Wilson
Subject: RE: Watching Issues

Kevin

Thanks for the tip. As it happens I am the project lead on virtually all of the product projects. As such, I get the update notices all of the time. What I want the filters to do is to allow me to quickly generate a predefined list of issues that I can then review online or quickly generate related reports. The key word is 'quickly'. I can get the same reports by opening each issue then printing it. That takes too long.

> ----Original Message----
> From: Kevin Wilson
> Sent: Thursday, October 13, 2005 9:58 AM
> To: Steve Erler
> Subject: Watching Issues
>
>
> On your dashboard there is a predefined filter called "Watches".
> If you watch an issue (click "watch it" in the issue side menu)
> the issue will be added to your watch list. Then to view all the
> issues you want to keep tabs on in a single list click the Watches
> filter on your dashboard.


Nick Menere [Atlassian] added a comment - 14/Oct/05 02:22 AM
Kevin,

Are we able to close this as a duplicate of JRA-7068?

Cheers,
Nick


Kevin Wilson added a comment - 14/Oct/05 09:16 AM
I wouldn't say it is a duplicate since I am requesting that an advanced search interface be implemented that allows you to build your own queries so you can query all data that you can't in the current search module. But if there is no chance you would actually do that then yeah, you may as well close it. I'll comment in 7068 too but the key search you plan to implement needs to be cross-project not single project as you noted.

Anton Mazkovoi [Atlassian] added a comment - 16/Oct/05 06:17 PM
Thanks for the update. We will leave this issue open then to track the 'advanced query' support. I must admit that we are quite busy at the moment, and it will probably take us some time to implement this.

Kevin Wilson added a comment - 22/Feb/06 09:24 AM
Any chance this advanced query module could be in version 3.6?

Anton Mazkovoi [Atlassian] added a comment - 27/Feb/06 11:59 PM
Kevin,

I hate to disappoint, but we will not be able to look into this for JIRA 3.6. For JIRA 3.6 we are hoping to:

  • Simplify the project administration, as it is getting very difficult to use
  • Allow custom notification events to support custom workflows, such that it is possible to control which parties get e-mails for each workflow transition. At the moment one is limited to built in events.
  • Rework JIRA's searching internals such that there are more efficient, and hence reduce the amount of memory a search requires.
  • Allow to edit activated workflows as it is a huge paint at the moment. One needs to copy workflow, edit it and then migrate projects to it one by one.
  • other (over 100) minor improvements currently scheduled for 3.6

Unfortunately, this leaves very little space other large features.

Thanks,
Anton


Christian Foisy added a comment - 11/Apr/06 09:47 AM
I wholeheartly agree with Kevin that the current search interface is too weak. Simple things like links cannot even be selected. For example all the following issues JRA-3101, JRA-5560, JRA-7772, JRA-3923, JRA-6444. JRA-5152 are different manifestion of the lack of querying on issue links (maybe we should add their respective votes to this issue

Please help us keep our managers happy.


Anton Mazkovoi [Atlassian] added a comment - 13/Apr/06 03:30 AM
Christian,

I have linked these issues. Thank you for the information and your feedback.

We are trying our best to implement the popular feature requests as fast as we can, however there are quite a lot of them.

Thanks,
Anton


Kevin Wilson added a comment - 13/Apr/06 10:36 AM

Rework JIRA's searching internals such that there are more efficient, and hence reduce the amount of memory a search requires.

Wouldn't reworking the searching internals now without doing/accounting for the necessary code to support advanced queries cause double the work down the line?

None of the improvements scheduled for 3.6 will improve the productivity of jira users more than the implementation of this feature would. Searching has been the scourge of jira since its first release and many paying customers have been clamoring for its improvement for years now.

Please Please Please just implement this feature and shut us all up about the searching.


Christian Foisy added a comment - 13/Apr/06 01:07 PM
I understand that implementing a UI to support "complex" queries is likely to please just a few but still be inadequate for many if not worst because they like the current one,

May I suggest you look at the Fisheye (note: it uses Lucene also) search system. In short you do simple search with selectors and simple text fields, but all other requests use a mini-SQL called EyeQL. No heavy UI work involved, simple text field for query entry, a small parser to map input text query into a another internal one, render found issues in navigator. Give a filter name to this query. Bingo!

For sure I don't know how JIRA architecture makes the introduction of this feature easy or difficult, but it looks like a simple elegant solution to the complex query problem.


Kevin Wilson added a comment - 13/Apr/06 01:35 PM

...because they like the current one,

I do not advocate changing the existing search UI at all but rather add a link called "Advanced Search" at the top that takes you to the expanded search interface. This will allow simple queries to be handled as usual but also allow for deep digging when necessary.

I would suggest that the advanced search do direct db queries, people will understand if it takes a little longer to exec.


Anton Mazkovoi [Atlassian] added a comment - 18/Apr/06 02:33 AM
Thank you for your comments!

We indeed are thinking about introducing a query language to ensure that advanced queries can be executed. The query language has the added benefit of allowing to execute an arbitrary search through remote API (SOAP/XML-RPC).

Currently, I do not have a concrete implementation date for this feature, but advanced queries is something we realise is very important and are hoping to address in the future.

Anton


Christian Foisy added a comment - 18/Apr/06 09:49 AM
Thanks Anton for this good news.

Is there anything we can do so this project gets higher priority and a release version?


Kevin Wilson added a comment - 19/Apr/06 10:45 AM
(Granted the following will sound a little whiny, I apologize in advance but I had to bring it up.)

I hope this particular issue doesn't suffer the same fate as did JRA-699 which only took _3 years to make its way into jira._ (JRA-699 also mentioned the need for complex queries.)

Any chance this feature gets into jira within the next couple/few months? My managers have to decide whether we re-up our maintenance contract the end of June (or source another tracker .. eek!) and I must admit that decision rests heavily on when this particular issue will get into jira. They have been very patient for eight months now and it will be quite hard convincing them to pony-up $2k for another year without this feature coming down the pipe in the very near future.

I like jira at the issue worker level but managers make the decision whether to pay or flee and since they are the ones doing the searching, organizing, scheduling and reporting it would do Atlassian well to throw them a bone real soon and increase your sales to boot.

Just my $0.02


Anton Mazkovoi [Atlassian] added a comment - 27/Apr/06 06:34 PM
Christian,

At the moment I do not think there is anything I can suggest.

Thanks,
Anton


Morgan Filipsson added a comment - 06/Aug/07 10:36 AM
We are changing issuetracker from from Trac to Jira. In Trac there is a possibility to write any SQL query and display the result. I really miss that feature in Jira. As a Jira Administrator that would give the possibility to setup some basic queries for all users.

I have searched for a Plugin with that feature, but I have not found any. Is there anyone with such a plugin?


AAgave Maketime added a comment - 08/Sep/07 02:30 AM
>> I have searched for a Plugin with that feature, but I have not found any. Is there anyone with such a plugin? <<

At the risk of coming across as a spammer - yes, my company has just created one. We're currently looking for a handful of beta testers - - if you're interested, please shoot me an email and we can set you up.

http://confluence.atlassian.com/display/JIRAEXT/MakeTime


Kevin Wilson added a comment - 12/Sep/07 10:15 AM
What are your licensing terms and costs for the product, and what are its system requirements? e.g., Does it depend on Windows OS components or will it work in any browser on any OS? We may be interested in beta testing your product here at Comtrol if the licensing terms are reasonable.

Sandor Palfy added a comment - 26/Oct/07 06:36 AM
Hi,
I'm just another user who really needs this fetaure into Jira. SQL as the query language would be fine, no need any fancy HTML interface for it.
Also, don't restrict these into one level, we need to create queries like: list all the issues which has subtasks with certain criteria, etc. (We need JOINs)
Advanced filters should behave the same as regular ones, you should be able to save them, share them, create permlink, do bulk operations, etc.

As for the implementation:
The easiest way for this would be to use virtual tables or views what you can use in the query. These could be filtered based on permissions.

A basic list of the tables would look like:

  • Issue: all the standard and custom fields, both standard and sub tasks witha column referencing to parent task.
  • Linked issues table
  • Users
  • Groups
  • Group - User assignment
    etc.

With these, all you need is just standard SQL.

Regards,
Sandor


Sandor Palfy added a comment - 28/Nov/07 05:50 AM
Is there any progess with this issue?