I do this a lot. It makes a handy speed-controlled buffer, especially using small wheels for detailed work that would be awkward on the bigger wheels. The chief objection is that you'll get your lathe bed covered in compound crap, and need to clean it off regularly because spattered brown compound dust will get into the gears of your tool post (assuming a metal lathe).