The description of this sounds like worn or loose valve guides to me. There shouldn't be any significant amount of left/right (or radial, is probably a better term) play of the valve guide in its channel. One way to test this is to just take the valve guide out and work the valve without it there. If the noise you're hearing goes away, then the problem is with the valve guide (or the channel).

You can do similar sorts of tests by borrowing a felt from one valve and using it to double up on another. if the noise goes away with the doubled felt, that suggests maybe you're using the wrong thickness of felt.

But remember: Just getting the noise to go away isn't your complete goal. That result is only good if it's done in the context of the valve alignment being correct.

Diagnosis by internet can be difficult.