Exposed filters wouldn't work for this kind of thing, nor would any pre-packaged module that I know of. For passing the arguments I would say that you need to create a small custom module that creates the needed form element and redirects to the url of the panel with the appropriate arguments added at the end. This can be accomplished with the forms api.
I don't know if I understood all of what you are trying to accomplish, such as whether you are wanting to build in the ability to narrow down the selection with more and more criteria, or if the user is just selecting one criteria per click that changes the whole page. Multiple criteria narrowing by terms with arguments is tricky and may require a lot of custom coding. As far as I know there isn't a module set up that does it right out of the box. Sending single arguments is fairly straight forward if you are using the advanced taxonomy blocks module.
