Issue Details (XML | Word | Printable)

Key: JRA-7909
Type: Improvement Improvement
Status: Open Open
Priority: Blocker Blocker
Assignee: Unassigned
Reporter: Claudia Goldgruber
Votes: 293
Watchers: 142
Operations

Add/Edit UI Mockup to this issue
If you were logged in you would be able to see more operations.
JIRA

Search/ filter for "empty" fields

Created: 09/Sep/05 04:02 AM   Updated: Thursday 06:14 AM
Component/s: Filtering & Indexing
Affects Version/s: 3.2
Fix Version/s: 4.0

Time Tracking:
Not Specified

File Attachments: 1. Java Source File AllParameter.java (3 kB)
2. XML File atlassian-plugin.xml (1 kB)
3. Java Archive File customfield-search-plugin-1.0.1.jar (5 kB)
4. Java Archive File customfield-search-plugin-1.0.jar (5 kB)
5. Java Archive File customfield-search-plugin-1.0.jar (5 kB)
6. Java Source File EnhancedFreeTextSearcher.java (3 kB)
7. Zip Archive jra_7909_3_11.zip (14 kB)
8. File project.properties (0.1 kB)
9. XML File project.xml (4 kB)
10. Text File SystemInfo.txt (1 kB)

Environment: 3.2.2 Enterprise Edition, Build #90 (please refer to the attachment for system info)
Issue Links:
Blocker
 
Cloners
 
Duplicate
 
Reference

Participants: Alex Kwarts, Andrew McLaughlin, Anna Berns, Anton Mazkovoi [Atlassian], Bettina Zucker, Brian Lane [JIRA Product Manager], Brian Williams, Chris P., Claudia Goldgruber, Clay Selvey, Cédric de J., David Feldman, Doug Dixon, Eyal Kaduri, Jeff Turner [Atlassian], Jennifer Marienfeld, Joanna Thurmann, John M. Black, Kai B., Mariano Benitez, Marilyn Daum, Mason Freed, Matt Doar (Consulting Toolsmiths), Megan Sumrell, Mehmet Ali Bektaş, Michael Brändel [Falk Marco Polo Interactive], Nancy Belser, NY User, Olle Friman, Randy Lukashuk, Ray Oei [Furore], Richard Rauser, Sami Merovuo, Stephane Lecuyer, Tom Furrer, Tom Urban and Vishwajeet Singh
Since last comment: 9 weeks, 2 days ago
Labels:
Support reference count: 30


 Description  « Hide
Our customer's main problem is the following:
How can we filter all Standard Issue Types (or Sub-Task Issue Types), which do not have an Original Estimate yet?

But there are other scenarios where we miss the possibility of searching for fields "that are still empty"
e.g., find all issues where there is no "Fixed in Build" (Text Field (< 255 characters) with Free Text Searcher)
or find all issues where there is no "Release Note" (Free Text Field (unlimited text) with Free Text Searcher)



 All   Comments   Work Log   Change History      Sort Order: Ascending order - Click to sort in descending order
Jeff Turner [Atlassian] added a comment - 09/Sep/05 06:25 AM
Yes, it's an unfortunate limitation in the search system. For now, a workaround is to sort by the relevant field. The first issues in the search will then be those with empty field values.

Claudia Goldgruber added a comment - 09/Sep/05 07:18 AM
We already apply this workaround, but would need to have the search system enhanced.
Is there a feature request on your roadmap that I can vote for? I imagine that we are not the
only JIRA users/ customers requesting this type of search.

Jeff Turner [Atlassian] added a comment - 11/Sep/05 07:30 PM
I couldn't find an existing issue. I've changed the type of this issue to record it.

Andrew McLaughlin added a comment - 11/Oct/05 11:05 PM
I second the motion for allowing one to filter for a NULL Date Due value. I have several shared due date filters set up so staff can see what past due, due this week and due beyond next week. It would be great to be able to have a saved filter to also see what items do not have a due date.

Marilyn Daum added a comment - 14/Jun/06 01:47 PM
I would also like the ability to search for a blank text field, e.g. Bugnote Description.

Bettina Zucker added a comment - 21/Jun/06 07:49 AM
Hello!

I'm also missing the possibility to search for empty fields.

If I search for a jira system field of type user, for example "assignee", I can search for an empty value by selecting "unassigned".
The same kind of search for a custom field of type user is not possible! Why not?
There are 2 serachers for custom user fields, but no one works like the searcher used for jira system fields.

By the way it is not nice in Jira that the custom field searchers are so often different from the system field searchers.
This inhomogeneity makes Jira less well usable.

Cheers
Bettina Zucker


Tom Furrer added a comment - 22/Jun/06 01:40 AM
Also required is to be able to filter on the system value "none" for select lists

Olle Friman added a comment - 30/Jun/06 08:44 AM
We have added a couple of custom fields for release handling. The information in these fields is of interest for certain parties, but only if they are filled in.
We would then want the ability to find all issues where these fields are "not null".

Tom Urban added a comment - 12/Jul/06 08:42 AM
I add my name to the list of those requesting the ability to filter on blank/null fields. BTW - thx for a very nice product. Kudos.

Kai B. added a comment - 26/Oct/06 02:50 AM
We also need the ability to filter for empty fields.

Irene


Stephane Lecuyer added a comment - 26/Oct/06 08:34 AM
We also need to be able to filter on empty fields. Maybe you could add a checkbox beside every field called <Select if NOT NULL> ?

Ray Oei [Furore] added a comment - 27/Oct/06 03:35 AM
Yep. Searching on fields, including custom, which are NULL of NONE would be a big improvement

Mehmet Ali Bektaş added a comment - 26/Dec/06 07:27 AM
This functionality is very important to quality assurance proccess. In order to ensure that some special information is supplied correctly there must be some controls. There must be an oportunity of finding issues in which required informations have not been filled. So it will be possible to find people who doesn't conform to standarts. That's why I want to be able to know in which issues the particular fields are empty or null.

I know there is a workaround solution but as you know it is not effective, it is very cumbersome. For example currently, when I want to obtain a list of related issues I reach the results in 2-3 minutes in the event that I am lucky. But if there was such a utility with which I can filter issues according to their EMPTY-NULL fields it would take just a few seconds. Sometimes I have to consider two or three parameters at the sametime, so it takes even much time.

Morover an extra bandwith is needed to download the unnecessary issues and there is an extra workload on the server. As a consequence solution of this problem will bring us a great convenience.


Cédric de J. added a comment - 20/Feb/07 04:11 AM
Agreeing with most people here. This is sorely needed. In my case, I more specifically need to be able to search for dates which are either null or in a given range.

Anna Berns added a comment - 22/Mar/07 07:37 PM - edited
Agree that being able to filter on "blank" (or NULL or "" or whatever the syntax would be) is crucial – we'd like to create filters with this for various fields (e.g. "needs estimate", "needs priority", "unassigned" etc).

Eyal Kaduri added a comment - 19/Apr/07 05:47 AM
I second ( actually 15th )
this request,

I've been asked numerous times for this and could not provide a proper solution
Sorting the table doesn't help that much .

Voting for it as well..


Joanna Thurmann added a comment - 22/Jun/07 02:57 PM
I am adding an enthusiastic and hopeful (???) vote #90 to this feature. We are sorely missing the ability to search for empty fields and especially 'unassigned' option for custom fields of type user picker or multi user picker. For example, we have a "QA Lead" field of type user picker and need to find all issues which are not yet assigned to a QA Lead. It's impossible right now, aside from the above cumbersome "Find All Issues and Sort by that Column" workaround. Please work on this.

Richard Rauser added a comment - 03/Jul/07 05:57 AM
I'm shocked to learn that this functionality is not included in Jira. This is a serious oversight in my view.

Sami Merovuo added a comment - 19/Jul/07 03:59 PM
We are waiting for this feature as well. And btw, that workaround is poor...

For example, we have a policy that developers have to "analyze" all received bug reports by giving working estimates for them. Since you cannot create a filter where you list all issues that are UNRESOLVED and not estimated, this cannot be controlled in a simple way. This would be a very simple query to do in SQL, I just tried it. So could you please add this feature also to the JIRA client?

We are currently using Jira Enterprise Edition, Version: 3.9.1.


NY User added a comment - 20/Aug/07 12:46 PM - edited
Hopefully additional comments and voting will help this issue be resolved soon! It is an important reporting capability to be able to create views where information has not been populated in custom fields.

For those looking for a different type of work around:

1. Make it a mandatory field up front
2. Make it a drop down field (not always possible)
3. Populate the field with default text
4. VOTE TO HAVE THIS CORRECTED!


John M. Black added a comment - 09/Oct/07 03:19 PM
While the symptoms described here are slightly different, I believe solving JRA-1560 would address all the needs expressed here.

If all 57 people watching this Voted for JRA-1560, it would move up to 4th place.


Megan Sumrell added a comment - 19/Nov/07 07:42 AM
PLEASE resolve this issue!!

Clay Selvey added a comment - 19/Nov/07 09:34 AM - edited
disregard

Matt Doar (Consulting Toolsmiths) added a comment - 27/Nov/07 04:56 PM
A text searcher that supports searching for "none"

Matt Doar (Consulting Toolsmiths) added a comment - 27/Nov/07 05:11 PM - edited
I've just attached 5 files that provide a new searcher type for text fields. If you choose the new Enhanced Free Text Searcher type for a custom text field, and then reindex, you can use the search term "none" as part of your query and only issues with no content in the field will be returned.

Atlassian developers: this searcher adds an artificial value to every document with an indexable value. The query is a boolean of the Lucene MatchAllDocsQuery AND NOT artificial_value as suggested in the Lucene FAQ

Apart from the potential performance issues with large numbers of issues, does this approach have other drawbacks that I am missing?

Usage

To use the files, first read up about creating your own plugins, then create a directory structure like:

project.xml
src\etc\project.xml
src\java\plugins*.java

Compile with maven 1.x as usual, and deploy the resulting jar file. This has been tested with Jira 3.10 but probably works with earlier versions (Lucene 1.9+ is needed I think).


Matt Doar (Consulting Toolsmiths) added a comment - 27/Nov/07 05:14 PM
This is necessary to build the plugin

Matt Doar (Consulting Toolsmiths) added a comment - 27/Nov/07 05:15 PM
This is the compiled jar file for the plugin.

Matt Doar (Consulting Toolsmiths) added a comment - 27/Nov/07 05:16 PM
I've made it easier for people to try this idea out. The latest attached file customfield-search-plugin-1.0.jar provides the new search type.

Bettina Zucker added a comment - 28/Nov/07 07:35 AM
I just tried out the .jar with jira 3.9.1, it does not work with that version.
I'll check it again when doing the next jira upgrade.
Bye
Bettina Zucker

Matt Doar (Consulting Toolsmiths) added a comment - 28/Nov/07 10:44 AM
Hi Bettina,

Just out of interest, what was the failure with Jira 3.9.1?

~Matt


Bettina Zucker added a comment - 29/Nov/07 01:55 AM
Hello Matt,

here is the error log of my test yesterday:

2007-11-28 14:32:39,547 main ERROR [atlassian.plugin.parsers.XmlDescriptorParser] There were problems loading the module 'customfield-searcher'. The module and its plugin have been disabled.
2007-11-28 14:32:39,547 main ERROR [atlassian.plugin.parsers.XmlDescriptorParser] There was a problem loading the descriptor for module 'customfield-searcher' in plugin 'Custom Field Search'.
Class version is incompatible with current JVM: plugins.EnhancedFreeTextSearcher
com.atlassian.plugin.PluginParseException: Class version is incompatible with current JVM: plugins.EnhancedFreeTextSearcher
at com.atlassian.plugin.descriptors.AbstractModuleDescriptor.init(AbstractModuleDescriptor.java:73)
at com.atlassian.jira.plugin.JiraResourcedModuleDescriptor.init(JiraResourcedModuleDescriptor.java:45)
at com.atlassian.jira.plugin.customfield.CustomFieldSearcherModuleDescriptor.init(CustomFieldSearcherModuleDescriptor.java:35)
at com.atlassian.plugin.parsers.XmlDescriptorParser.createModuleDescriptor(XmlDescriptorParser.java:147)
at com.atlassian.plugin.parsers.XmlDescriptorParser.configurePlugin(XmlDescriptorParser.java:85)
at com.atlassian.plugin.loaders.SinglePluginLoader.loadPlugin(SinglePluginLoader.java:86)
at com.atlassian.plugin.loaders.SinglePluginLoader.loadAllPlugins(SinglePluginLoader.java:50)
at com.atlassian.plugin.loaders.ClassPathPluginLoader.loadClassPathPlugins(ClassPathPluginLoader.java:50)
at com.atlassian.plugin.loaders.ClassPathPluginLoader.loadAllPlugins(ClassPathPluginLoader.java:63)
at com.atlassian.plugin.DefaultPluginManager.init(DefaultPluginManager.java:70)
at com.atlassian.jira.plugin.JiraPluginManager.start(JiraPluginManager.java:28)
at com.atlassian.jira.ComponentManager.quickStart(ComponentManager.java:358)
at com.atlassian.jira.ComponentManager.start(ComponentManager.java:336)
at com.atlassian.jira.upgrade.ConsistencyLauncher.contextInitialized(ConsistencyLauncher.java:31)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3763)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4211)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1013)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:718)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1013)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442)
at org.apache.catalina.core.StandardService.start(StandardService.java:450)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:709)
at org.apache.catalina.startup.Catalina.start(Catalina.java:551)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:294)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:432)
Caused by: java.lang.UnsupportedClassVersionError: Bad version number in .class file
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:1815)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:869)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1322)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1201)
at com.atlassian.plugin.util.ClassLoaderUtils.loadClass(ClassLoaderUtils.java:40)
at com.atlassian.plugin.impl.StaticPlugin.loadClass(StaticPlugin.java:172)
at com.atlassian.plugin.descriptors.AbstractModuleDescriptor.init(AbstractModuleDescriptor.java:46)
... 28 more
2007-11-28 14:32:39,563 main ERROR [atlassian.plugin.parsers.XmlDescriptorParser] There were errors loading the plugin 'Custom Field Search'. The plugin has been disabled.
2007-11-28 14:32:46,374 main INFO [atlassian.jira.upgrade.ConsistencyCheckImpl] Starting JIRA Enterprise Edition, Version: 3.9.1-#234
2007-11-28 14:32:46,374 main INFO [atlassian.jira.upgrade.ConsistencyCheckImpl] Configured to use database: mysql
2007-11-28 14:32:46,452 main INFO [atlassian.jira.upgrade.ConsistencyCheckImpl]

*****************************************************************************************************
JIRA 3.9.1 build: 234 (Enterprise Edition) started. You can now access JIRA through your web browser.
*****************************************************************************************************
...


Matt Doar (Consulting Toolsmiths) added a comment - 29/Nov/07 11:14 AM
Looks like the jar file was built with jdk1.6 instead of 1.5. I'll upload a
1.5 version as well

~Matt


Matt Doar (Consulting Toolsmiths) added a comment - 29/Nov/07 11:15 AM - edited
Go to Manage Attachments and use the jar file uploaded on 11/29 if you are using jdk1.5 for Jira

Bettina Zucker added a comment - 30/Nov/07 05:28 AM
Hello Matt,

thank you for uploading the second .jar!
This one works for me (yes I'm using jdk 1.5).
I configured one custom field of type "Free Text Field (unlimited text)" to use the Enhanced Free Text Searcher and did a reindex.
Now I can search if the content is empty by giving "none" in the search field for this custom field.
This is really great help!

Bettina


Matt Doar (Consulting Toolsmiths) added a comment - 30/Nov/07 11:57 AM
Glad it worked for you. Now we just need some comment from Atlassian about the wisdom of this approach.

Jeff, Anton, or Ian: please would someone chime in here.

~Matt


Anton Mazkovoi [Atlassian] added a comment - 02/Dec/07 11:56 PM
Hi Matt,

I had a brief look at the classes. I think the approach is good.

Please note that the plugin will not work on JIRA 3.11 as Lucene has been upgraded to version 2.2, where Field.Text and Field.Keyword have been removed. It should be easy to get around this by using the constructors of the Field object, see the Lucene Javadoc.

As you have mentioned, using this plugin on a large set of issues could result in bad performance.

As far as I can tell, another affect of this is searching for "customfield_xxxx", where xxxx is the id of the custom field, will return all issues with the custom field having some value. (I have not actually tested any of this).

Cheers,
Anton


Matt Doar (Consulting Toolsmiths) added a comment - 04/Dec/07 03:56 PM
The source for the plugin for Jira 3.11

Matt Doar (Consulting Toolsmiths) added a comment - 04/Dec/07 03:57 PM
The plugin jar file for Jira 3.11

Matt Doar (Consulting Toolsmiths) added a comment - 04/Dec/07 04:01 PM
Anton, thanks for the response. Does Atlassian have a different approach in mind for implementing this feature?
I nearly ended up writing a plugin to provide raw SQL read-only access to get this working.

I've attached a zip file with the source and a version 1.0.1 jar file for the plugin that should work with Jira 3.11 in case other people want it.

~Matt


Anton Mazkovoi [Atlassian] added a comment - 04/Dec/07 10:19 PM
Hi Matt,

At the moment I do not have a detailed technical spec for how we are planning to do it.

My personal thinking is adding a flag to the Lucene document when the field is empty e.g. customfield_xxxx_empty: true. This will involve changing how the search sub-system works in JIRA, which will take some time. We will need to assess if this solution is viable when we schedule this feature.

Cheers,
Anton


Matt Doar (Consulting Toolsmiths) added a comment - 05/Dec/07 11:09 AM
Fair enough. BTW, the performance of the MatchAllDocs was better than I
expected. Selecting 2K from 18K issues took less than 2s for me.

~Matt

On Dec 4, 2007 8:20 PM, Anton Mazkovoi [Atlassian] (JIRA) <


Anton Mazkovoi [Atlassian] added a comment - 06/Dec/07 05:44 PM

Selecting 2K from 18K issues took less than 2s for me.

Thanks for the update. I expected it to be slower than this too.

However, 2s is fairly taxing. Lots of JIRA users have a lot more than 18k issues. We also rely on searching being sub-second so that we can display search counts for saved filters on various pages.


Matt Doar (Consulting Toolsmiths) added a comment - 06/Dec/07 05:51 PM
Yes, it's probably worth testing the different approaches to see where the
breakpoints are.

~Matt

On Dec 6, 2007 3:45 PM, Anton Mazkovoi [Atlassian] (JIRA) <


Michael Brändel [Falk Marco Polo Interactive] added a comment - 21/Feb/08 01:49 PM
This is definitely one of the most important feature I am missing in JIRA.

Nancy Belser added a comment - 20/Mar/08 09:24 PM
We need the ability to search for text values that are not empty. We have this feature in bugzilla and the lack of it in Jira may prevent us from switching products.

Doug Dixon added a comment - 24/Apr/08 09:28 AM
I also want to be able to search for issues where a certain field is set to any value (i.e. !=null)

BTW just noticed... this feature request is nearly 3 years old!!!


David Feldman added a comment - 10/Jun/08 08:13 AM
This one is also a problem for me. I have to create a dummy value for certain fields, make it the default, and then make the field be required. This seems to work, but then I have an issue dealing with the older tickets that existed prior to the workaround. I have to search for those using the filter-and-sort trick, then update those issues to use the dummy value for the field. It ends up being a whole lot of extra work.

Brian Williams added a comment - 10/Jul/08 05:40 PM
I would like to be able to search for "null" and also search for "not null" on custom fields.

Randy Lukashuk added a comment - 13/Aug/08 08:02 PM
One vote for me.

Jennifer Marienfeld added a comment - 21/Oct/08 01:49 AM
We also need the possibility to search for empty or not empty fields.

Vishwajeet Singh added a comment - 02/Jan/09 05:55 AM
Is there any time line decided on this issue??

Any update will be good for us.


Brian Lane [JIRA Product Manager] added a comment - 12/Jan/09 11:42 PM
Relating this to JRA-1560 as we are going to allow isempty/isnull JQL queries to execute.

Mariano Benitez added a comment - 12/Jan/09 11:52 PM
I will be out of the office on a business trip. I will be back in the office on January 19th.

En este momento estoy fuera de la oficina. Estaré de vuelta el 19 de Enero.

Med venlig hilsen / With best regards / Saludos


Mariano Benitez added a comment - 12/Jan/09 11:54 PM
I will be out of the office on a business trip. I will be back in the office on January 19th.

En este momento estoy fuera de la oficina. Estaré de vuelta el 19 de Enero.

Med venlig hilsen / With best regards / Saludos


Mason Freed added a comment - 02/Mar/09 11:27 AM
Adding another vote for this. Not being able to search for an empty Version number using the version searcher really sucks. The workaround (searching for all, then sorting by the version field, then manually selecting all of the empty ones) REALLY sucks. Please fix!!!

P.S. This is the first time I've come to this site looking for a feature and didn't find what I was looking for!


Mariano Benitez added a comment - 02/Mar/09 11:32 AM
I will be out of the office until March 3rd. Please contact Bhagat Nainani for any urgent issue.

En este momento estoy fuera de la oficina hasta el 3 de Marzo. Por favor contactar a Eduardo Rubio por asuntos urgentes.

Med venlig hilsen / With best regards / Saludos


Chris P. added a comment - 09/Mar/09 10:41 AM
Please provide a solution or workaround ASAP!

Alex Kwarts added a comment - 30/Apr/09 02:14 AM
I would also like the ability to search for a blank text field, e.g. "Original Estimate" I want to know if all the issues have an estimate entered.