Arrow key problem

I often jog with the keyboard arrow keys rather than the mouse because I can feel the keys while leaning over the workpiece to see what is happening. But if the last thing I did with the mouse is move the jog speed slider, then jogging with the arrow keys will both jog and change the jog rate! This is really annoying. Technically the slider does have focus but clearly one does not want to jog and change jog speed at the same time.