Shadowmask mode is a lighting mode shared by all Mixed Lights in a Scene. To set Mixed lighting to Shadowmask, open the Lighting window (menu: Window > Lighting > Settings), click the Scene tab, navigate to Mixed Lighting and set the Lighting Mode to Shadowmask.
You also need to select the desired Shadowmask mode to use from the Quality Settings (menu: Edit > Project Settings > Quality):
2017–09–18 Page amended
Shadowmask modes added to Quality Settings in 2017.1 NewIn20171