Forgotten Realms or Nentir Vale?
When it appeared in 4th Edition, I didn't take "Nentir Vale" as a "setting" like the Realms or Eberron, but as an example of what a setting could and might look like if designed under the base assumptions of 4th Edition, primarily:
The World is a Fantastic Place
The World is Ancient
The World is Mysterious
Monsters are Everywhere
In the 4th Edition DMG, the Nentir Vale is just the surrounding area of the example "base town" of Fallcrest. Yes, some of the nine base adventures that were published for the edition took place there (when they took place in the world at all), but I took the Vale as something that could be (and was intended to be) plunked down into just about any setting. I've placed the Vale in Eberron before and I bet people have placed it in the Realms.
Anyway, I recommend not using the Realms unless both you and your players are on the same page about the lore, or they're willing (and able) to learn about it as they play. Info dumps are notorious for slowing the game down and being misunderstood when they're not simply forgotten. You'd probably wind up having to make up a lot of stuff anyway, just to accommodate the things the players did that wouldn't make sense in the Realms.
That said, while there is a lot of lore available about the Realms, I have always taken the Realms as really just representing a setting in which it makes sense for everything from the core of a particular edition to be present, so that a DM could come up with any kind of adventure and allow any type of character they wanted and it would fit. Lore is great if one likes it, but hopefully is something that works with one's game, rather than hindering it.
Anyway, you say above "I'd have to make more stuff up on my own, though." That's not necessarily true. Consider working with your players to fill out the setting, as the game comes across new parts of it. In my experience, when a group collaborates on a setting or scene, everyone cares more about the details and remembers them better.
One of the primary advantages of the "points of light" setting is that no one in the setting really knows anything for sure. The World is Mysterious and Ancient and most people don't travel much, because Monsters are Everywhere. Civilized settlements are "points" of light, because they flicker and go out, and spark suddenly to life. A map might point to a village, but when the PCs get there, there's nothing but some haunted ruins. Or they might come across a village that wasn't on their map, and was completely unknown despite being only a dozen miles from the nearest other town. The point of the concept is to give the DM that freedom, to make it plausible that there could be an unexplored dungeon within a day's ride of town.
Good luck.