Looks like you're on Fedora Silverblue (or other Atomic version). This is happening because the system groups are in /usr/lib/group rather than /etc/group and this causes the issue you're seeing here. You can work around it by getting into a root shell with something like
sudo -i
and then getting the group added to /etc/group with
grep -E '^dialout' /usr/lib/group >> /etc/group
after that, you'll be able to add your user to the group with