To sum up the way they write, a rollerball writes with a thick, vivid line. The line may smudge if you quickly run your hand over it (as it uses liquid ink) but it is a smoother pen. A Ballpoint pen generally writes a thinner, less vivid line but it does dry instantly on paper.
If you're not sure what kind of pen you write with is, it's probably a ballpoint...most pens are. But you can always do a quick test keeping the above points in mind. Both pens use the same basic mechanism, where a rolling ball at the tip of the pen is 'inked' by a reservoir above it. As you write over a page, the inked ball rolls over the paper, leaving its mark.
The main difference is the ink inside that reservoir. The ballpoint uses a viscous ink, so it's not pure liquid. This is good news in some ways but bad news in others. Because of the ink formula, it dries instantly on paper and there is no smudging. This expedient little feature (like most expedient features) is really the main reason that ballpoints are so popular. And economy of course.
But because of the ink formula, the ball is not very responsive and a lot of people find ballpoints 'scratchy'. The ink can 'skip' too...because it is not a smooth flowing, liquid ink that is running over the ball, you can end up writing with a 'naked' ball, so no ink is left on your paper. We've all had this happen! Also, the ink can dry up in the reservoir. This is why those very cheap ballpoint pens stop working even though they look like they are full of ink.
Rollerballs on the other hand use a liquid ink. Or a gel ink. We'll treat them differently here. A liquid ink rollerball basically puts a greater amount of ink onto the paper than a ballpoint. 3-4 times the amount is an instructive standard and this is why your rollerball line is so vivid and also why some rollerballs can smudge.
But they are much more smooth. This makes writing easier than it is on a ballpoint pen and also just makes your work look a bit more alive! Personally i really like a rollerball line over a ballpoint line. Rollerballs can cause a couple of problems if you are using poor quality paper. As it is a liquid ink, it can 'bleed' or 'feather' if you are using poor paper. Just grab some Rhodia and you'll be fine!
Gel ink rollerballs are the same as above but even more alive still! The nature of gel as a medium allows greater pigmentation of the ink and this just leads to a greater range of possible colours, rendered with a bit more vigor.
There are, of course, exceptions to these guidelines. Fisher Space Pen ballpoint pens, due to their refill, experience none of the heartaches advised above for ballpoint pens generally. Likewise, any rollerball, such as Retro 51 or Diplomat, which uses the Schmidt Easy Flow 9000 refill will dry instantly and doesn't smudge!
Pens aside, writing really is a marriage of paper and ink and so your paper is really important. No matter your pen, if you're using poor quality paper you will run into problems. Likewise, if you have Rhodia or Clairefontaine paper (in particular...there is plenty of great paper out there), your pen will write like a champ!