Set the raycasts to either detect or not detect Triggers.
A Collider can be set up to act as a trigger which will detect other Colliders entering its volume but won't physically collide with them. Often, it is desirable to avoid detecting trigger Colliders with raycasts because they don't represent solid objects. This property lets you choose whether or not raycasts should detect triggers.
Additional resources: Collider2D.isTrigger, Raycast.