I’m not sure the correct way, but I will share what works well for me. Over time, I’ve fine tuned my process and I have a 99% success rate with rivets.
The thing that made the biggest difference for me was using a piece of 1/4in thick aluminum to put under my PCB/Rivets before hammering. I install the Rivets, put the aluminum on top and flip it. Before hammering, I tape the PCB to the aluminum with painters masking tape so PCB doesn’t bounce and rivets don’t fall out. Have a look at the picture to see how nice the rivets turn out (and how flat they are).
I make my holes a tight fit with the rivets .7mm drill for .4mm Rivets and 1.6mm drill for 1mm rivets. I find even going to a .10mm larger results in problematic rivets.
I found the actual hammer makes a huge difference. This Hammer - Nylon end for .4mm Rivets, and Copper end for 1mm. I’ve never overdone it with this hammer.
Lastly, I put the board by my ear and run my finger over all the rivets. If they are not in right you can hear a very quiet click when you run your finger over.
Hope this helps.