I'm having difficulty recording hiss-free voice audio using my PC using a USB headset combo, at least I think the problem is my headset because the built-in speaker gives much better quality, but the recordings aren't very loud and have brief, ocassional fuzz sounds. What is the least expensive type of microphone that would give hiss-free recordings.
The problem is not really going to be the mic in your headset, although obviously a cheap mic is likely to be lower quality than a good one.
The problem is that recording in to a normal PC is at a very low voltage, so the difference between noise (mains hum, interference etc) and the signal you want is not that great, so when you amplify the signal you also amplify the noise, leading to hiss/buzz/hum.
If you want to record at better quality, the single best thing you can do is add a proper soundcard to your PC - and by proper, I mean a powered, external sound card. This gives you a much improved amplifier and lower noise floor, and gets the signal up to a decent voltage (typically line voltage) before it gets to your PC, which means it is much more resistant to interference.
Many external sound cards will also let you use XLR balanced connections to microphones - which are much more resistant to interference.
It isn't just the signal to noise level, but also generally the isolation of the sound processing hardware. On cheap, consumer sound cards, the audio processing is generally done on circuitry that is impacted by other things going on in the computer. This variation results in noise in addition to the normal signal to noise issues inherent with just about any consumer grade mic (particularly that goes through a 1/8th inch jack).
The only way you are likely to get hiss free audio is with a prosumer level capture device that has balanced inputs (XLR) to deal with the signal to noise issues and a power conditioned and isolated analog to digital converter. You will also need a decent XLR based mic. (Personally I highly recommend getting a used SM58 for a "cheap" quality option. They are nearly indestructible so getting a used one should be ok.)
Cheap is relative though, I'd expect the total cost would still be in the neighborhood of $200 or more for all the pieces you'd need. Another option, though a little more expensive, is to pick up a dedicated recorder like a Zoom H4N that will let you record stuff without the computer. They run a few hundred dollars without the mic.
An external sound card and a good phantom-powered XLR mic is certainly the pro-audio way to solve this problem. However, I've had good experience with USB audio products for casual recording.
I use a Blue Snowball for Skype meetings. The Snowball is a bit quieter than the mic on my macbook pro, but I can move it away from my mac's fan noise, so the overall quality is much better. One of my friends prefers the Logitech G330 headset, and that also produces good results. I've experienced hiss problems with other Logitech products on my Mac, but then Logitech doesn't (or at the time didn't) deliver mac-specific drivers for their gaming devices.
There are plenty of podcaster-oriented USB microphone products that will be more than adequate for recording hiss-free voice. You really only need to take the soundcard route if you plan on recording multiple sources at once, or need more power than the USB bus can provide.
The "fuzz" sounds may be your breath blowing across the mic. You can either speak across, rather than into, the mic or you can look for a better windscreen and/or pop filter.