there's a lot of good advice above, but at the end of the day it's all down to what you want to do!
if you want good quality recordings that you wont feel embarrassed about when you play them to others [on chordie...hopefully] your best bet is to get a multi-track digital recorder, new one's can be expensive, but there are most likely used one's out there.
multi-tracking means putting different sounds onto separate tracks, without affecting the sound on each, therefore you can mix the tracks when you've finished recording to get a balanced sound, with possibly one guitar in the left channel, another guitar [or keyboard] in the right channel and your vocal track in the middle.
you can also add harmonies, bass, drums etc, all at different times and independently.
overdubbing simply means recording eg a voice over a pre-existing guitar track. with this method you cant remix at a later time, so you have to make sure the mix is correct before starting.
as already mentioned above, you can get a recorder that will interface with your pc, so you may be able to transfer your tracks onto audacity to mix down and add effects, check russels tutorial in the sticky section above.
if you really want to get into recording it can get very expensive and complicated, so beginners kit would be;
a recording device.
a good mic, [i use a beyerdynamic stage mic through an alesis mixer].
i prefer using a guitar pick-up as i always end up knocking the mic or over or under recording, or picking up the sound of people walking out side or cars passing.
some recorders have a built in metrognome, but if yours doesnt, you may want to get a drum machine to keep the beat.
that's it, your basic starter kit, or if you prefer you could down-load audacity [it's free] then get a decent mic [ensure you have a mic interface on your pc before lashing out your hard earned]
and follow russel's instructions.
happy recording, and let's hear them when their done