My personal preference is NES -> Master System -> Atari due to the game libraries.
With that said, do you have any favorite games that are exclusive to either console? That might help make the decision. Are you able to gettl the games? I would argue without games to play, it probably isn't worth it. Do you have the means and ability to repair or make needed upgrades? These systems are getting on in years and I find myself repairing my consoles more frequently as the years go on.
If that is the case, I would go with your favorite library of games, and if not that, then go with the console that has your favorite game. I recommend the NES not only because it has an amazing original library, but there are also still some small niche studios making games for the NES today and it is amazing what they have learned to squeeze out of that hardware.
Looks like a couple people mentioned this, but you will likely want/need to mod whatever system you get. I have a modded NES and Atari 7800, so I can comment on those.
There used to be a NES mod that fixed the pin connector issue, called "Blinking Light Win" https://www.arcadeworks.net/products/blw?variant=36483581116569. They've been sold out for at least a couple years (I've sporadically checked out of curiosity since I bought mine). But if you can get your hands on one I strongly recommend it. It has made my NES incredibly reliable and it's very easy to install. No soldering, just open the console, remove the 72 pin connector, and pop in the BLW. Done.
For the Atari 7800, there's a composite video/s-video board mod you can pick up. I'm forgetting what the name of it is, and I don't have the manual handy at the moment, but I will try to update later with the name and link. Anyway, this one is more involved and requires some soldering. It's not too difficult, just requires some patience, and it's very worth it. Composite audio is a separate solution, but doesn't require extra hardware, just a little more soldering.
I hope that helps! Feel free to reach out about these mods and I can explain further.
As someone that owned all three, the NES without question. If you have budget or desire for a second pick, my recommendation would be to try and find a Sega Game Gear and pick up an adapter for playing Master System games.
Without any hesitation: the NES. Easily the best library out of all 3 systems, most intuitive game design, plus you can't beat that feeling of inserting the cartridge like a VCR.
I think you should find games first. No point in owning an NES without finding something to play on it. But if you are looking for opinion on the hardware itself.
I own all 3 systems.
Atari 7800: isn't fare to compare it to a NES or Master System. Think of it as a suped up 2600. Great system if you use the European controller or a Sega 9pin controller. Game selection is good but limited. Best way to play 2600 games. Rf only sucks but can be modded. I'd get it if there are 2600 or 7800 games you wanna play. But the Atari 50th collection has pretty much everything you'd want to play on it.
NES: is a good system for the time, but hasn't aged well. Getting games to work has turned into a ritual for me. But when they work you'll have access to the best games from the late 80's and early 90's. Controller has aged like fine wine. Most of its non-nintendo library can be found elsewhere for cheap while the Nintendo games are behind a subscription service. Not many games exclusive these days but worth playing anyways
Master System: the NES we have at home. Mostly a similar library of games to the NES but are sometimes better, sometimes worse. Depends on the publisher. In north america it was forgotten and game selection was limited. Controller is mushy but a genesis controller can fix that.
Out of the 3 the master system is probably the best experience, but the NES has the best games.
Mushy controller and not as good a library as NES, whose controller aged like fine wine. Why is the Master System in your opinion the best system out of the 3 and not the legendary NES? Why are you having so much trouble getting games to work?
In North America the NES isn't a top loader and due to the mechanism they used the pins ware out, especially after 35-40 years. On top of that there is very little in the NES Library that's only on the NES these days. If I hadn't already invested in an NES and games, I'd just get the NES Classic and be done with it. If I wanted extra games I'd get collections like the Castlevaina Collection, The Mega Man one, and Disney Afternoon. You can extract the ROMs from these collections and put them on the NES Classic if you are willing to fiddle.
Meanwhile the master system is often times forgotten, but has a similar library for third party games. And what unique games does have are rarely found elsewhere as the master system version. Let alone the European releases like Sonic the Hedgehog.
With that said, I prefaced this with "Find games first" for a reason. The master system has a smaller library of games, and if OP was looking for games more like Dragon Quest Warrior or Final Fantasy, then a system with only Phantasy Star would be a bad choice.
From my experience arcade ports and titles released for both are usually noticeably better on the Master System in both looks and sound. Rampage is one example. It also has less issues with the cartridge port. One caveat is the controller ports are prone to bent pins. Otherwise NES all the way.
If you're after a vintage collection, I believe 7800 would let you build up the most the fastest and cheapest - the library isn't huge, and it doesn't have the same demand as the Nintendo/Sega stuff. You could likewise go for an Atari XE for breadth since that lets you access the 400/800 platform's game library, and the white-pastel case design of those models makes them a good showpiece.
If it's emulation, the options are too prolific to name - you could plug in a USB gamepad to just about anything and get a really accurate experience. People do seem to like having a "game playing box" though, and that leads in the direction of an RPi, Anbernic, Miyoo or MiSTer.
If you are in this more for the idea of hacking on retro-styled projects there are a batch of new single-board computers pairing the vintage CPU with a modern microcontroller - the two I would look at for this are Agon Light and Neo6502. Agon has been demonstrated running MSX game code natively on its eZ80, emulating just the I/O parts: Link. Agon could feasibly run Master System games(the video output is limited to the exact same color palette), while Neo6502 might be able to pull this off with the Ataris or NES.
I haven't seen anybody mention failing systems. These consoles are all old enough that some TLC may be required soon to replace failing capacitors, etc. If you're good with that and the (light) headaches involved in connecting old consoles to new TVs, there are solid recommendations in this discussion.
If you're okay with emulation, the Anbernic RG351MP (or the newer 353) will play games from Atari 2600 through PlayStation 1, including arcade machines and old DOS games. The Anbernic hardware is very nice and exceptionally portable.
Steam Deck and Nintendo New 3DS are also phenomenal emulators for old games, but require some setup.
Not sure how you feel about emulation, but take a look at FPGA systems like the MiSTer or Analog Pocket. These use hardware/chip simulation to recreate a 1:1 real hardware experience. You can also use original controllers via adapters with 0 lag, so the games feel the same too. With one of these systems you will be able to play all of those systems plus a bunch more. It’s a cool system, highly recommended.
MiSTer is a great option for high accuracy and zero/low latancy emulation. And besides being able to use original controllers you can also drive an analog display with an analog IO board, if that's desirable to you.
If you want real hardware, this isn't it. And I can understand wanting the real hardware. But for a nearly perfect recreation of the experience without the headaches of maintaining aging hardware, MiSTer does the job.
Analog Pocket looks cool too. But wanted to nerd out on the MiSTer specifically.
Apologies if you know all this stuff already; I just like talking about it:
NES has more genre creating games than any console in history. PlayStation 1 was a similarly seminal thing, but NES was when the deep magic was written. A lot of its games simply had no predecessors; they were invented from first principles.
The exact choice of titles will depend on your enjoyment level for games that are pretty unpolished from today’s POV, but many of them are good enough to still hold up even now.
History factor: Metal Gear, SMB1, Double Dragon, Metroid, Castlevania
History factor but also still fun: Contra, Legend of Zelda, SMB3
Speaking of Contra - there is a new Contra out March 12, 2024 (today! or tomorrow depending on timezone), Contra: Operation Galuga. I haven't played the game yet, but the trailer looks amazing!
Like an actual, original console? If so, I'd say Master System. The 7800 sucks compared to the 2800 and the NES is so incredibly popular it doesn't seem as special as a Master System would be.