Hide field from customer on search panel

Hi, is it possible that you can add a functionality of hiding a custom field in search panel but only for some users (groups of users)? Or if not is there any workaround/trick ? We don't want customers to filter by some attributes, but we would like to have a possibility for that. What we want to hid is our own custom field developed in a plugin. Is it possible at all that you are working on such things or the only possibility is to do it using plugins, velocity files etc... thanks in advance
best regards