Any chance you can get it to a Vet?
Are you supplimenting/dusting the worms?
Is it a worm only diet?
Will he eat while being held? This way you could help him get the worm started.
It sounds nerological to me but a Vet. visit may be needed.
Best of luck, keep us posted.