I've started to incorporate lights more into my projects. Before I wouldn't even bother and just use whatever available light was there.
Before for white balance I would just use the settings like Daylight, Fluorescent, Cloudy, etc. But I'd like to start using the kelvin settings along with my lights.
For example, the lights I have are balanced for I think 6500 Kelvin according the little manual the lights came with. Does that mean I would go into my camera and set it to 6500?
Theoretically yes, though in practice this may not give you a perfect white balance. It really depends on how accurate the light source is that you are using and the camera as well. If the source in question is an LED then chances are it will be quite a bit off from whatever the manual says. The only way to know for sure is to test and see.
Sounds like you may be using a Canon DSLR or C series camera, you can also do a manual white balance on those which will probably give you the best result quickly. The other option is to use gels like CTO, CTB, Minus Green, and Plus Green on your light to match it by eye to natural daylight. This will require some experimentation on your part.