Restrict Computer Logons to a Group

We have 2 PCs that we need to block logins for everyone except a couple of users.  I have those users in a Security Group and tried applying a Group Policy to an OU with those PCs that allows them and Domain Admins "logon locally" rights.  I can still login with a test user that is not in the group.

Do I also need to apply "deny logon locally" for this to work?

Is there an easier way?