Taxonomy+Views+Panel

Exposed filters wouldn't work for this kind of thing, nor would any pre-packaged module that I know of. For passing the arguments I would say that you need to create a small custom module that creates the needed form element and redirects to the url of the panel with the appropriate arguments added at the end. This can be accomplished with the forms api.

I don't know if I understood all of what you are trying to accomplish, such as whether you are wanting to build in the ability to narrow down the selection with more and more criteria, or if the user is just selecting one criteria per click that changes the whole page. Multiple criteria narrowing by terms with arguments is tricky and may require a lot of custom coding. As far as I know there isn't a module set up that does it right out of the box. Sending single arguments is fairly straight forward if you are using the advanced taxonomy blocks module.

The user should select from the dropdown list(that is an exposed filter working now)
but I have a problem, I could not set put my Content into a panel or another separate place, everything is in 1 page View now.

Views and exposed filters are working with 1 step only. If there is only 1 Taxonomy category then I would not need any panels or other stuff. But in my occasion, there are 2 taxonomy categories related to 1 content item.

When I open the View Page, I see this:
http://i45.tinypic.com/fepflt.jpg

When I click to Summarized Taxonomy term(Amsterdam), I get this(Check out the url):
http://i48.tinypic.com/t018ua.jpg

Then I try to change the Filters but it doesn't take effect
http://i49.tinypic.com/t6y0l2.jpg
The url stays the same

What do you recommend?

I can't really diagnose what is going on with your particular view if there is a filter not working, however, like I said exposed filters probably won't work for what you are trying to do.

You have to use real arguments. Passing arguments through panels to a view is tricky but it is doable. Passing exposed filters to a panel is not at all possible, so if panels are supposed to enter into this equation at any point in the project then you need to start with arguments from the start.

http://i46.tinypic.com/1z4ityw.jpg

Now I get the taxonomy term id's as urls, how can I modify my View so that I can use a separate column(list of Taxonomy term ids) to use instead of Exposed filters?