You shouldn't have any problems shuffling.
Skullbriar, the Walking Grave doesn't have its counter-keeping ability while face-down in exile, since face-down exiled cards have no characteristics, so even if it somehow kept its counters when being put into that face-down pile (which, to be clear, it shouldn't), it would immediately lose them upon being returned to the battlefield.