Totally agree, and it's a great idea to invest this time in other stuff to learn more and get other things done.
Even if you do those things that benefit your company however, in theory at least you should have reported back that your task is finished, and you'd usually be assigned with what's next on the priority list... So still the temptation is always there.
Well maybe your honesty follows a stricter standard than mine. :-)
It's a gray area, but if my task touches module A and I spend some extra time cleaning up module A, or writing tests for it, my conscience is clear. Or if I finish a deployment script but think reading a Chef tutorial might teach me a better way of doing it next time, sure. The work may not be finished just because "it runs." I wouldn't fill up 4 hours with that kind of thing, but 1, sure (depending on the urgency of other tasks, etc.).
I guess it also depends on your company culture and your boss's/client's expectations. I have clients who would get upset if I billed an extra half hour, and other clients who want me to spend that hour on tests/refactoring.
Even if you do those things that benefit your company however, in theory at least you should have reported back that your task is finished, and you'd usually be assigned with what's next on the priority list... So still the temptation is always there.