Re: Best System for a horror game
If the emphasis is investigation, I would recommend the Gunshoe system which is used in the Trial of Cthulhu RPG. The rule system was designed specifically for solving mysteries, but it still handles combat well enough.
Another good option is Palladium's house system which is used in horror games like Nightbane, Beyond the Supernatural, and Dead Reign. It's sort of a very combat-centric derivative of D&D. Palladium also has very nice random insanity table, the best this side of Call of Cthulhu RPG.
The Storyteller system from White Wolf games is a standard and holds up well to this day. It's skill heavy so it handles investigations well and also combat well but doesn't stand out in either category. This is a system which a lot of gamers know already, so it's a solid choice for getting players set up quickly. Storyteller characters tend to get hurt quite a lot, but they are difficult to kill outright.
Personally, I like C.O.R.E. (Chupa Open Roleplay System). I have run a couple of different Cthulhu scenarios with this ruleset, but almost no one has heard of that.