It must remain.
It must remind you of missing out on music. That might get you to work slightly harder through the day and squeeze in a few hours for writing that song you’ve been wanting to write.
For this, the job might hate you, and that hate might get you to love music more. You’ll be more focused on music, and discipline and a routine will set in for nurturing the music in you.
Once the music gets sense and soul into you, you know it’s time to quit. But you’ll have to wait for that time to come. Keep that passion for music burning. Be patient; good music will meet a good musician!
For me, before I got to produce music full-time, I kept quite a few daytime jobs, the most interesting one as a door-to-door internet connection salesman.