engine:
It does if they want it to. People roleplay their characters not knowing things the player knows all the time.
There's a difference between roleplaying while knowing more than the character, and roleplaying when you already know what the plot twist is supposed to be. In a genre that is about said plot twists, it either fails flat or you're not really playing that genre.
engine:
The latter, primarily. Lots of mystery stories are like that, at least in part: at some point, they either know or think they know what the actual situation is and the mystery is about how to prove it to everyone else. I know the king's advisor is evil, but no one believes me without evidence and if I just stab him them I'm the bad guy.
If it is the advisor, what you have is that either you switched genres mid-play, or perhaps to a different mystery (instead of puzzling who did it, puzzling how it was done in order to prove it).
Not that because the players think the advisor did it means the advisor really did it. Done right with the intention to keep things ambiguous enough, the players won't even agree on who the bad guy is during most of the mystery section of the adventure, each trying to prove their own pet theory.
engine:
Rather a fun mystery that everyone's bought into than a "proper" mystery that turns out to be a disappointment.
So your argument is that it's better to ignore an entire genre on the chance you screw up as a GM? because for it to be mystery, it
has to have enough internal consistency and a logical chain of evidence that the reveal can be deduced by a sufficiently clever audience, it's a genre about solving the puzzle.
If there isn't a solvable puzzle, then it isn't, by definition, mystery, it's simply another genre dressing as one.
engine:
It's only one aspect of the charm, and if the reveal turns out to be blah (as it is in many mysteries) or if hiding details requires too much blocking then it's a setup for disappointment.
Well if you fail as a GM or if the group can't get into the mood for the game's genre then of course the game fails. If it were a collaborative work and the foe is just as blah, I assure you the game would fail as well.
The group, of course, is free to change their minds and switch or alter the foe, but the sole GM is just as able to change the foe with none the wiser if it looks like it'll be a disappointment.
If the foe only turns into a disappointment when the final confrontation happens (say, a really lucky roll or someone came up with an inventive solution that oneshots it), then neither the group effort nor the single GM has any chance of salvaging that.
engine:
That's the entire point: If the players are involved and collaborating then there's no fridge logic to worry about. People don't go around poking holes in their own fun ideas.
I've had more than enough fridge logic moments of my own plots to believe I wouldn't do it if it were a collaborative work, I've had players say the next day or session "why the hell did I do that? that didn't make sense", I've had players make plans for their characters to do X or Y, only to suddenly change their minds, I've had players change their minds on what they want their character to do in the future every other hour, I've had players talk to me to kill a character and start over with a new one because they don't like where they progressed the character towards, or because they gave it this or that mannerism and they discovered they suck at portraying it or it grew annoying.
To say that a collaborative game would magically make everyone not poke holes on their own (or others) ideas or make them not change their minds is folly.
engine:
Players know and figure out things, and will play along even when they do to make a cool situation work. If players aren't wrecking a plan it's not necessarily because the plan is working, but might be because the players want it to work and are blocking themselves before the GM has to.
If it happens, it happens, just as often they'll come to the wrong conclusion, or not all players come to the right conclusion and they'll get their "I knew it"/"told you so!" satisfaction. Even if they all figure it out, if as a GM you did a good enough work and made the puzzle challenging enough to solve, the players feel good because they outsmarted the challenge.
If you missed and made the puzzle too easy, you can always add new evidence that puts the previous conclusion into doubt or shift genres, granted, this is easier to do correctly between sessions or on a slower format like pbp, but you can prepare a few things you can throw at the players to stall them while you figure out what to do.
engine:
No, it also works when the players who know about it want it to work.
Then it's no longer the hidden foe, just a foe. Not that regular foes aren't useful or interesting, mind, but they are different things. Compare the movie where you get a scene every so often about what the bad guy is doing to the movie where the real bad guy is only revealed at a dramatically appropriate time.
In order to do the later, unless there was absolutely no hint that the real bad guy existed, you need a certain level of internal consistency on the hints that were given, and you can only achieve that correctly when it's planned that way.
Just to be clear, I'm not against full on collaborative efforts, just on the idea that they magically fix everything or their use on genres that rely on a twist, puzzle or secret to work. Now, partial collaborative efforts can work on those, but they face the same 'the players did x and derailed everything' problems.