Advancing the Shader Tree

The Shader Tree is both immensely powerful and tremendously flexible. It is this flexibility that can lead to it feeling unwieldy at times. Based on user feedback, it was a design requirement to allow materials and textures to be applied at any level in the scene, even globally if necessary, but as the saying goes, "With great power comes great responsibility".

As a result of this flexibility, diligence is required to keep the Shader Tree organized, and the effort necessary increases as the complexity of a project increases. Understanding how to work with layers in the Shader Tree and what each contributes is an important start in wielding the power that the Shader Tree provides. The basics are covered in Shader Tree, Shader Tree Item Layers, and Layer Effects. Beyond the basics covered on these pages, it was important to add some additional information going over some more advanced aspects of shading surfaces in Modo.

Groups and Groups

The number one most important item in the Shader Tree is the Material Group item. In the Add Layer function of the Shader Tree, it is simply referred to as a Group, but here we're calling it a Material Group to avoid any confusion with actual groups. The Material Group serves multiple purposes and understanding its few settings are extremely important in getting the most out of them.

When you add a Material Group item to the tree, they are simply empty containers. These containers can hold any number of layers, including collections of other groups. In this case, they are strictly organizational devices, providing hierarchy to a scene. For example, you might associate similar materials to an item in the scene, making it easier to navigate a complex tree. This can be illustrated by selecting a few layers in the tree and pressing the Ctrl+G keyboard shortcut, grouping together similar materials under an undefined Material Group item.

The Material Group item also provides the principal means for applying materials to surfaces using masks. By selecting from named items and/or polygon tags in the Material Group item's Properties panel, any contents (or layers) within the Material Group can be limited to only affect geometry surfaces that carry that specific tag in the scene. This is done automatically every time you apply a Material Tag using the M keyboard command. Pressing M opens the Polygon Set Material dialog, allowing you to define a name and pressing OK generates a new Material Group item in the Shader Tree, automatically setting the Tag Type and Tag Name options to the particular name that was defined. It's a very simple procedure, but the implications of the power that the Material Group provide are immense.

Masking Surfaces

While the Material Group is an easy way to assign surfacing for geometry when rendering, its reliance on polygon tags limits what is possible with just Material Groups. Not all surface borders are hard-edged polygonal boundaries. Modo provides several ways to mask textures that provide much finer control. The masking functions combined with the layering possibilities of the Shader Tree makes it simple to produce very complex surfaces with ease. Whenever an alpha transparency mask is applied to a texture layer, the underlying layers are revealed.

Group Masks

The Group Mask is a Layer Effect option, defined for a layer in the Effects column of the Shader Tree. It allows the contents of an entire Material Group, regardless of complexity or number of layers, to be masked by using any single texture item. The texture can be a procedural texture such as a Noise Procedural layer, an image map, or even a weight map using the Sorry you didn't find this helpful layer. The use of image maps provides very fine pixel-level control. You can create these images externally using an image editor, or paint them directly in Modo using the built-in paint tools. The use of gradients opens up a number of interesting possibilities as well, where Material Groups can be masked by the numerous gradient input parameters, such as slope or incidence angle.

Note:  You can render complicated material setups directly in the Advanced 3D viewport and make changes in real time. The Advanced 3D Viewport supports Group Masks allowing you to mask multiple material layers.
The Advanced 3D viewport supports group mask Blend Mode and Opacity changes, resulting, in most cases, in little difference between the representation of a material in Advanced 3D viewport and the representation of a material in Modo’s Preview Viewport.
For more information, see 3D (OpenGL) Viewport.

Layer Masks

The Layer Mask result is similar to the masking functionality of the Group Mask, but its application is limited to a single layer within the Shader Tree, providing even more granular control. There are two ways to apply a layer mask:

• Choose the Layer Mask effect and place the layer directly below the layer you want to mask, or

• Drag and drop the layer onto the target layer, automatically defining it as a layer mask effect.

You can modify the settings of the texture layer, toggling the visibility of this kind of mask using the + icon preceding the layer's name.

Alpha Channels

Modo opens and respects embedded alpha channels or layer transparency on imported image map layers. Their actual application can be a point of confusion, because some artists are surprised when they can't see through their surface in the transparent areas of the image map. This is because Modo respects the alpha channel for the image map as a texture layer (revealing any lower surface shading), but it doesn't serve double duty as Transparency (or Stencil/Dissolve) for the surface when set as the default Diffuse Color effect. To achieve these effects, you would also need to apply the image as a Layer or Group Mask, or additionally assign the Alpha channel on the image as Transparency or Dissolve using the Alpha Only option in the image's Properties. Alternatively, you can simply use the RGBA layer effect to designate both diffuse color and transparency (applied as Stencil) using a single texture layer.

By virtue of how Modo handles alpha transparency for a image maps, this makes it possible to layer any number of transparent images and combine the result into a single complex surface. Even when painting, using transparent image layers, Photoshop-like multi-layered images are possible by stacking several image map layers with Alpha channels. You can select the layer in the Shader Tree to designate which image receives the brush strokes while painting, leaving other layers unaffected.

Item Masks

Within the Material Group item's Properties panel, you can specify item masks in addition to the other settings. Item Masks can be extremely useful limiting the application of images or shading to a single Mesh Item layer, without resorting to a bunch of different polygon tags. This makes it possible to apply different shading or image maps to an instance simply by specifying the instanced item's name in the Item property of the Material Group item. Instances usually have the inherited parent items name followed by a number in parenthesis. A simple Item selection shows which Instance is the target for the mask. Item Masks also allow shading to be applied to individual Items that would otherwise overwrite other same named (tagged) surfaces in the scene.

Layering Surfaces

Not only can you layer images and textures in a single material, you can also layer different tags for interesting results. Polygons can only have a single part or material tag, but they can have both tags simultaneously as well as belonging to an unlimited number of selections sets. The tagged polygons do not have to be identical in their grouping, or even similar. It is therefore possible to create a selection set that spans polygons from two separate material surfaces, define it as a mask and apply an image that straddles the two surfaces. Use a straight material tag for the two regular surfaces and then assign the smaller areas to the selection set with custom UVs. It's also very useful to apply labels and such.

Cleaning Up the Shader Tree

On occasion, you may find the Shader Tree encumbered with unnecessary layers that need to be removed, but choosing which layers to delete can be difficult. These extra layers can be the result of experimentation or excessive copying of material presets.

The Purge Unused Materials command removes all Shader Tree material and/or texture layers that do not contribute to the final rendered image, including those that reference unassigned polygon tags. These could occur when geometry is deleted, or new tags are assigned, as polygons can only hold a single material or part tag. Moving item layers into a scene from another can also cause extra layers to be added.

To activate the purge, select the command from the menu bar, under Texture > Purge Unused Materials.

Targeted Control

One often confusing aspect of the Shader Tree is its Base Shader item that defaults to the top of every Shader Tree. The Base Shader controls visibility, shading rate, and many other aspects of the rendered image. In a way it is like a soldier at the gate controlling what information gets by the gate to the renderer and how much of that information is passed along for evaluation. Due to the global nature of the Shader Tree, with the Base Shader at the top and unmasked, it has meant that any shader items added below it have their settings overwritten by Base Shader settings unless the lower level shader was masked appropriately and placed higher up in the Shader Tree.

The juggling of Shader Tree layers and Group Material masks evaluating based on their position can get confusing, especially in complex scenes. Sometimes you are just looking for a simple way to make a single object invisible to the camera. To do this, Item Shaders have been introduced. They are in every way identical to the regular shader items except that they are the only Shader Tree layer that isn't represented in the Shader Tree. These are assigned and modified directly in the Item List. When any of their Control options are enabled, those settings always override those of the Shader Tree counterpart upon evaluation.

To assign an Item Shader, in the Item List, right-click on the target Mesh Item layer and select the Create Item Shader option from the menu. With the Item Shader selected, you can modify its attributes in the Properties panel. To target a specific control, such as Visibility, disable all the Control options except for Control Visibility, and then disable the single Visible to Camera option. This makes the targeted item invisible to the camera, yet it still casts shadows and shows up in reflections. Item Shaders simplify many aspect that used to be difficult to apply.

Some Additional Notes

It's easy to forget that texture items such as image maps and procedurals can be added to lights and environments as well. By adjusting their Projection types and Effect settings, a whole range of interesting effects are achievable. Texture layers can be used to mask any other types of layer, including Surface Generators and Fur. The only layer that cannot be used as a mask for Surface Generators or Fur is the Occlusion layer. This is because it requires rays to be fired to calculate and this only happens at render time, which is too late in the evaluation process to produce the intended result. In these cases, bake the occlusion layer out to a image map texture and then reapply the image map in place of the occlusion.