Can I reject a review assignment?

Yes. On the review start page, click Reject Review and provide a reason. You will be removed as a reviewer for that workspace. If other reviewers are assigned, they can still complete the review.