I've done both. I've taken pencils and a sharpener to a live game. I've since switched to a quality Rotring Mechanical pencil to eliminate the need for a sharpener (you'll always need a quality eraser).
I've also rolled with a few, archival quality pens like this set of thin (.3, never more than .5) Staedtler Pigment Liners that come in a convenient case that claps shut. Some of the colors are more useful than others, so I usually buy singles and double up on what I use most often (blue, black and red).
It is true that it's easier to have your ideal set up at home, and this is more frequently the case these days as getting to a game is a luxury of time and money I don't have.
I've mostly seen people (including broadcasters) color code hits, walks, Ks and runs typically.
My goal is to always produce a clean, yet detailed card that is legible and easy to decipher. My best advice is to try and achieve this either with a normal (i.e., not colored) pencil or a single color pen/pigment liner. If you're able to do that consistently, then adding color when you're comfortable will make your card look even better. All the colors in the world won't make a poorly kept scorecard easier to read or look any better.