To remove a layer from a layermask, use the logical AND operator on the original layermask and the bitwise complement of the layer to remove it.
originalLayerMask |= ~(1 << layerToRemove);