Welcome! Lots of us work full-time jobs while building up our VA practices. It can be done - just be prepared for exhaustion.

(Hmmm... reminds me of having a new baby, almost!).
I think the current economy will actually strengthen the VA industry because more people are going out on their own (and don't necessarily have the admin skills to handle day-to-day business mgmt) and more small businesses are finding it tough to keep employees when they don't have the work to keep them busy! Having a VA means being able to utilize our services only when needed...
Good luck w/your endeavor!