This assumption is exactly correct.
My pgp keys are symmetrically encrypted at startup, and unlocked with my keyring.
Physical, USB keys exist for this purpose.
Personally - I'd lose a dongle.
All communication, of every kind, between any people, should be signed and encrypted with...