PixelIop: Getting Started with Image Processing¶
The PixelIop class provides a specialized version of Iop, adding boilerplate code for convenience of use at the expense of limiting potential use cases. PixelIop inherits the majority of its functionality from the base image processing class Iop, so if you haven’t already, please read 2D Architecture before coming back to this section.
PixelIop is intended for image processing operators that are not spatial in any way - that is, each pixel in the output depends upon the pixel at the same coordinates in the input, and no other pixels. Examples of Ops that might be implemented as PixelIops could include Gain or Desaturate.
PixelIops are an excellent way to get started with using the NDK in NUKE, and have a very low memory footprint. If the algorithm you are looking to implement can be reduced to only rely on the input pixel or row, and thus be implemented as a PixelIop, then it should.
PixelIop:
is intended for use by image manipulation operators that, for each pixel in the output, rely solely on the corresponding pixel in the input (althought strictly speaking it can use any pixel in the corresponding scanline row).
adds simplified channel manipulation functions.
The PixelIop Class Specifics & Required Virtual Calls¶
PixelIop varies from Iop by having two new pure virtual calls that an Op built on it must implement. These are:
-
void
PixelIop::in_channels(int input, ChannelSet &mask)¶ in_channels is called by the PixelIop’s definition of _request, for each input. It passes the input number as ‘input’ and a reference to a ChannelSet as ‘mask’. This ChannelSet is initialized with the channels that have been requested - in_channels should fill in any other channels that are needed (or remove those that are unnecessary) on the given input.
For example, a simple Desaturate would implement in_channels like so:
virtual void in_channels(int input, ChannelSet& mask) const { if (mask.contains(Mask_Red) || mask.contains(Mask_Green) || mask.contains(Mask_Blue)) mask += Mask_RGB; }
That is, if any of the Red, Green or Blue channels have been requested, it needs to request all of them from its input.
-
void
PixelIop::pixel_engine(const Row &in, int y, int x, int r, ChannelMask, Row&)¶ The pixel_engine function is called with the row from the input already fetched with the correct channels from in_channels. Generally, the output row and the input row are actually the same Row, but this should not be relied upon. A simple implementation of pixel_engine that merely multiples all the pixels by 0.5 is as follows:
virtual void pixel_engine(const Row &in, int y, int l, int r, ChannelMask channels, Row& out) { foreach(z, channels) { const float* inptr = in[z]; float* outptr = out.writable(z); for (int x = l; x < r; x++) { outptr[x] = inptr[x] * 0.5; } } }
A PixelIop-derived Op can optionally choose to override the default _validate provided by Iop.
-
void
PixelIop::_validate(bool for_real)¶ The default, Iop-provided _validate merges IopInfo from all inputs into one, and turns on all the channels set in out_channels. By default, out_channels is set to Mask_All, but it can be set as you desire by overriding _validate, calling set_out_channels, and then calling the base class’ validate. For example:
void _validate(bool for_real) { set_out_channels(Mask_RGB); PixelIop::_validate(for_real); }
Another common situation is that the current interface settings mean your Op does not actually do anything to the underlying image data. In this situation, you should instead set the output channels to Mask_None in your _validate(), which allows NUKE to optimize the Op tree.
void _validate(bool for_real) { if(_doingAnyWork) { set_out_channels(Mask_RGB); PixelIop::_validate(for_real); } else { set_out_channels(Mask_None); } }
More _validate circumstances are described in the 2D Architecture section.
As with all Ops, PixelIops must additionally implement Description, and may optionally implement knobs() to provide user interface elements.
Getting Started: The Basic Node¶
Congratulations for wading through the theory thus far! Now for the start of the fun stuff - writing your first operator. Get yourself setup with a compiler as per Appendix A: Setting up Projects & Compilers and compile up the Basic example. We’ve replicated the Basic code below for convenience.
static const char* const HELP = "Basic: Does nothing but copy the input from input0 to the output";
#include <DDImage/NukeWrapper.h>
#include <DDImage/PixelIop.h>
#include <DDImage/Row.h>
#include <DDImage/Knobs.h>
using namespace DD::Image;
class Basic : public PixelIop {
public:
void in_channels(int input, ChannelSet& mask) const;
Basic(Node *node) : PixelIop(node) {
}
void pixel_engine(const Row& in, int y, int x, int r, ChannelMask, Row& out);
static const Iop::Description d;
const char* Class() const {return d.name;}
const char* node_help() const {return HELP;}
void _validate(bool);
};
void Basic::_validate(bool for_real) {
copy_info();
set_out_channels(Mask_All);
}
void Basic::in_channels(int input, ChannelSet& mask) const {
//mask is unchanged
}
void Basic::pixel_engine(const Row& in, int y, int x, int r, ChannelMask channels, Row& out){
foreach (z, channels) {
const float* inptr = in[z]+x;
const float* END = inptr+(r-x);
float* outptr = out.writable(z)+x;
while (inptr < END) *outptr++ = *inptr++;
}
}
static Iop* build(Node *node) {return new NukeWrapper(new Basic(node));}
const Iop::Description Basic::d("Basic", "Basic", build);
First, some notes about this plug-in from top to bottom:
The HELP string and node_help function do as the names suggest: provide help to the user in the form of a tooltip when the user clicks the “?” button in the node’s properties.
The in_channels function is required by the parent PixelIop class and tells NUKE what channels of the image we would like to access. In general, we would like all channels, and so leave ChannelSet& mask untouched. If we wanted, we could restrict the channels being operated on, for example, if we only desired the red channel: mask &= Mask_Red;.
The build(Node *node) function takes our image operator (Iop) and wraps it into a NUKE’s node for us via the NukeWrapper class.
The d(“Basic”, “Basic”, build) defines the class / Iop name used to create the node. It is vitally important that the first parameter string matches the filename of the compiled plug-in, that is, “Basic” refers to Basic.dll (on Windows), or Basic.so or Basic.dylib depending on your system.
NukeWrapper will be discussed in more detail in Working with NukeWrapper. For now, simply be aware that this is a convenience function that adds common controls and functionality to the operator. In this circumstance, it is responsible for adding the channel, mask, and mix controls you see when you open the node’s parameter panel.
The actual interesting things in this snippet are the _validate and pixel_engine functions:
Essentially, the _validate function tells NUKE about the size and format of the generated images, including the image channels we’re going to access and create. In this example, copy_info() is called, which takes the format information from the input image and copies it onto the format of the current “Basic” node.The set_out_channels(Mask_All); call simply says that we are looking at (and modifying) all channels in the image. The Mask_All could be replaced by Mask_Red or Mask_Alpha if we just wanted to look at the red or alpha image channels respectively.
Now, the pixel_engine function is where the pixel reading / writing happens. Notice that the class Row is used a lot in this function. This is because NUKE likes to operate on scanline rows of images at a time. This makes sense for many reasons, such as operating on different scanlines in parallel (multiple threads) or, more importantly, only needing to keep a handful of image rows in memory at any given time. The row index you are currently working on is given by y, with the offset into the scanline given by x. Note that either or both of these can be negative (for example, if the input image is outside the bounds of the frame rectangle). The last index of the scanline is given by r, and the width of the input scanline is therefore given by r-x-1. The channels that you have access to are set in the channels. As you can see from the code, in[z]+x and out.writable(z)+x give you pointers to the input and output row data for channel z, where z is obtained using the foreach function to iterate over the collection of channels passed in. The output pointer can obviously then be accessed to do what ever you’d like.
Have a good play around with this code and start making some changes. Take what you’ve learned and apply it to understanding the Add.cpp example shipped with the NDK. This is the actual source code for the Add node compiled into NUKE, and is a great starting point for exploring PixelIops. If you run into areas you’re not sure about, jump on to the next section which covers some further fundamentals, with reference to the more complex Grade example.
Building the Grade Node¶
Now, lets take a look at our first ‘real’ node: namely the Grade operator. The source code from which this node is actually built is shipped with the NDK, so again, compile yourself up a copy referencing Appendix A: Setting up Projects & Compilers. The source code is duplicated below for convenience.
// Grade.C
// Copyright (c) 2009 The Foundry Visionmongers Ltd. All Rights Reserved.
const char* const HELP =
"<p>Applies a linear ramp followed by a gamma function to each color channel.</p>"
"<p> A = multiply * (gain-lift)/(whitepoint-blackpoint)<br>"
" B = offset + lift - A*blackpoint<br>"
" output = pow(A*input + B, 1/gamma)</p>"
"The <i>reverse</i> option is also provided so that you can copy-paste this node to "
"invert the grade. This will do the opposite gamma correction followed by the "
"opposite linear ramp.";
#include "DDImage/PixelIop.h"
#include "DDImage/Row.h"
#include "DDImage/DDMath.h"
#include "DDImage/NukeWrapper.h"
#include <string.h>
using namespace DD::Image;
static const char* const CLASS = "Grade";
class GradeIop : public PixelIop
{
float blackpoint[4];
float whitepoint[4];
float black[4];
float white[4];
float add[4];
float multiply[4];
float gamma[4];
bool reverse;
bool black_clamp;
bool white_clamp;
public:
GradeIop(Node* node) : PixelIop(node)
{
for (int n = 0; n < 4; n++) {
black[n] = blackpoint[n] = add[n] = 0.0f;
white[n] = whitepoint[n] = multiply[n] = 1.0f;
gamma[n] = 1.0f;
}
reverse = false;
black_clamp = true;
white_clamp = false;
}
// indicate that channels only depend on themselves:
void in_channels(int, ChannelSet& channels) const { }
void pixel_engine(const Row &in, int y, int x, int r, ChannelMask, Row &);
virtual void knobs(Knob_Callback);
const char* Class() const { return CLASS; }
const char* node_help() const { return HELP; }
static const Iop::Description d;
void _validate(bool for_real);
};
void GradeIop::_validate(bool for_real)
{
bool change_any = black_clamp | white_clamp;
bool change_zero = false;
for (int z = 0; z < 4; z++) {
float A = whitepoint[z] - blackpoint[z];
A = A ? (white[z] - black[z]) / A : 10000.0f;
A *= multiply[z];
float B = add[z] + black[z] - blackpoint[z] * A;
if (A != 1 || B || gamma[z] != 1.0f) {
change_any = true;
if (B)
change_zero = true;
}
}
set_out_channels(change_any ? Mask_All : Mask_None);
PixelIop::_validate(for_real);
if (change_zero)
info_.black_outside(false);
}
void GradeIop::pixel_engine(const Row& in, int y, int x, int r,
ChannelMask channels, Row& out)
{
foreach (n, channels) {
unsigned z = colourIndex(n);
if (z > 3) {
out.copy(in, n, x, r);
continue;
}
float A = whitepoint[z] - blackpoint[z];
A = A ? (white[z] - black[z]) / A : 10000.0f;
A *= multiply[z];
float B = add[z] + black[z] - blackpoint[z] * A;
if (!B && in.is_zero(n)) {
out.erase(n);
continue;
}
float G = gamma[z];
// patch for linux alphas because the pow function behaves badly
// for very large or very small exponent values.
#ifdef __alpha
if (G < 0.008f)
G = 0.0f;
if (G > 125.0f)
G = 125.0f;
#endif
const float* inptr = in[n] + x;
float* OUTBUF = out.writable(n) + x;
float* END = OUTBUF + (r - x);
if (!reverse) {
// do the linear interpolation:
if (A != 1 || B) {
for (float* outptr = OUTBUF; outptr < END;)
*outptr++ = *inptr++ *A + B;
inptr = OUTBUF;
}
// clamp
if (white_clamp || black_clamp) {
for (float* outptr = OUTBUF; outptr < END;) {
float a = *inptr++;
if (a < 0.0f && black_clamp)
a = 0.0f;
else if (a > 1.0f && white_clamp)
a = 1.0f;
*outptr++ = a;
}
inptr = OUTBUF;
}
// do the gamma:
if (G <= 0) {
for (float* outptr = OUTBUF; outptr < END;) {
float V = *inptr++;
if (V < 1.0f)
V = 0.0f;
else if (V > 1.0f)
V = INFINITY;
*outptr++ = V;
}
}
else if (G != 1.0f) {
G = 1.0f / G;
for (float* outptr = OUTBUF; outptr < END;) {
float V = *inptr++;
if (V <= 0.0f)
; //V = 0.0f;
#ifdef __alpha
else if (V <= 1e-6f && G > 1.0f)
V = 0.0f;
#endif
else if (V < 1)
V = powf(V, G);
else
V = 1.0f + (V - 1.0f) * G;
*outptr++ = V;
}
}
else if (inptr != OUTBUF) {
memcpy(OUTBUF, inptr, (END - OUTBUF) * sizeof(*OUTBUF));
}
}
else {
// Reverse gamma:
if (G <= 0) {
for (float* outptr = OUTBUF; outptr < END;)
*outptr++ = *inptr++ > 0.0f ? 1.0f : 0.0f;
inptr = OUTBUF;
}
else if (G != 1.0f) {
for (float* outptr = OUTBUF; outptr < END;) {
float V = *inptr++;
if (V <= 0.0f)
; //V = 0.0f;
#ifdef __alpha
else if (V <= 1e-6f && G > 1.0f)
V = 0.0f;
#endif
else if (V < 1.0f)
V = powf(V, G);
else
V = 1.0f + (V - 1.0f) * G;
*outptr++ = V;
}
inptr = OUTBUF;
}
// Reverse the linear part:
if (A != 1.0f || B) {
if (A)
A = 1 / A;
else
A = 1.0f;
B = -B * A;
for (float* outptr = OUTBUF; outptr < END;)
*outptr++ = *inptr++ *A + B;
inptr = OUTBUF;
}
// clamp
if (white_clamp || black_clamp) {
for (float* outptr = OUTBUF; outptr < END;) {
float a = *inptr++;
if (a < 0.0f && black_clamp)
a = 0.0f;
else if (a > 1.0f && white_clamp)
a = 1.0f;
*outptr++ = a;
}
inptr = OUTBUF;
}
else if (inptr != OUTBUF) {
memcpy(OUTBUF, inptr, (END - OUTBUF) * sizeof(*OUTBUF));
}
}
}
}
#include "DDImage/Knobs.h"
void GradeIop::knobs(Knob_Callback f)
{
AColor_knob(f, blackpoint, IRange(-1, 1), "blackpoint");
Tooltip(f, "This color is turned into black");
AColor_knob(f, whitepoint, IRange(0, 4), "whitepoint");
Tooltip(f, "This color is turned into white");
AColor_knob(f, black, IRange(-1, 1), "black", "lift");
Tooltip(f, "Black is turned into this color");
AColor_knob(f, white, IRange(0, 4), "white", "gain");
Tooltip(f, "White is turned into this color");
AColor_knob(f, multiply, IRange(0, 4), "multiply");
Tooltip(f, "Constant to multiply result by");
AColor_knob(f, add, IRange(-1, 1), "add", "offset");
Tooltip(f, "Constant to add to result (raises both black & white, unlike lift)");
AColor_knob(f, gamma, IRange(.2, 5), "gamma");
Tooltip(f, "Gamma correction applied to final result");
Newline(f, " ");
Bool_knob(f, &reverse, "reverse");
Tooltip(f, "Invert the math to undo the correction");
Bool_knob(f, &black_clamp, "black_clamp", "black clamp");
Tooltip(f, "Output that is less than zero is changed to zero");
Bool_knob(f, &white_clamp, "white_clamp", "white clamp");
Tooltip(f, "Output that is greater than 1 is changed to 1");
}
static Iop* build(Node* node)
{
return (new NukeWrapper(new GradeIop(node)))->channelsRGBoptionalAlpha();
}
const Iop::Description GradeIop::d(CLASS, "Color/Correct/Grade", build);
Once again, notes from top to bottom. Standard HELP definition, set of includes and a namespace declaration, followed by a static declaration of the CLASS name, used in a variety of places throughout the remainder of the Op. This can be useful when developing, as it allows the class name to be easily altered during the development cycle - see the Versioning discussion for areas where this may come in handy.
We then declare our subclass of PixelIop, with member variables used to store values in use by the interface knobs. Often, depending on relevant coding standards, you’ll see such member variables prefixed with an underscore. We initialize these to their default values in the class constructor, and set up in_channels to define no extra dependency for an output channel than the corresponding input channel.
Next up, we have our _validate implementation. Rather more interesting than the basic example, it figures out if the current interface settings are going to make any difference to the image data on any of the four potential channels, if not calling set_out_channels with a Mask_None, otherwise with Mask_All. This has an identical effect to using disable() in that NUKE optimizes this operator out of the tree. It then calls the default PixelIop implementation, which merges all the input’s IopInfo structures into the output stream, in this case just copying through the single input’s structure. Subsequently, the op sets its IopInfo.black_outside flag dependent on whether the Op itself is altering the 0 level in any of the image channels affected. This has the effect of switching off repetition of the bounding box edge pixel out to the format size, in the circumstance the bounding box is smaller than the format and the tree has not had a black pixel padding inserted into the stream. This means that if you alter a comped element’s black point, it won’t edge repeat over a larger background element. See the IopInfo detail in the _validate section for more information on this mechanism.
The majority of the body of the operator is taken up with the pixel_engine, which is responsible for the color correction operation being applied to the pixels. We won’t take you through line by line; instead, we’ll introduce the fundamental functions utilized within. Breaking down the actual color correction operation itself is an excellent first exercise for learning to use a debugger, so if you’re a little rusty there, do take this as a good opportunity to do so. From this point forward, we will assume that you are able to hook up a debugger and inspect variables, insert breakpoints, and so on.
As in the Basic example, foreach is used to iterate across all the passed-in channels, however, in this circumstance we then test for the current channel’s index (an associated integer identifier). If the index is above 3 (that is, not Chan_Red, Chan_Green, or Chan_Blue), then we copy the data through untouched using the row.copy method and move on to the next channel in the set. row.copy is preferred in this circumstance over setting up a pointer on the source row and a writeable pointer on the target row, as in some circumstances it can simply switch the pointer over, avoiding the overhead of the data copy.
If the incoming channel is in the color channel range, then the engine sets up to do the calculation necessary to build the output row. It uses a shortcut for the circumstance where the input row is all zero and the current settings don’t involve changing the 0 value. This helps speed up processing on the common use case of 3D elements with poor bounding box settings and large regions of black. To do this, it uses row.is_zero(Channel) to, as you would imagine, identify if the row is all zero for the selected channel, and row.erase(Channel) to zero the selected channel.
As before, there’s the definition of the source and target pointers and the decision tree associated with the various Grade node settings. This is where the image processing actually happens - if you’re interested in the mechanics, feel free to hook up that debugger. An interesting point to note is the use of memcpy**s from input to output buffer. Note that if you call **row.copy instead, having already written other parts of your row, you may end up blitzing already calculated data.
After the engine functionality, we have the knobs(Knob_Callback) call. As mentioned previously, this is where your user interface is defined. In this circumstance, we have a number of AColor_knob entries, which appear as a slider that can be split four ways plus defined by a color swatch alongside the standard NUKE animation controls and a couple of Bool_knob entries that appear as checkboxes. As with the Basic example, you can also see in our build(Node*) method we use the NukeWrapper class to add channel, mix, and mask knobs. We also call the NukeWrapper.channelsRGBOptionalAlpha() method to preset our channel knob to be RGB by default.
Each knob call specifies:
the type of knob
the variable in which to store its value
the name
optionally the label that appears on the interface (if you want it to be different to the name)
a range to be shown (in the case of slider-based knobs)
a tooltip to be shown to the user on a hover event.
There are a vast array of options for knob types, interface layout, and so forth, as covered in the Knob Types and Knob Flags sections. Note that for the variable passed to the Bool_knob entries we use the address of a bool variable, whilst for the AColor_knob entries we use an array pointer. The array must have the correct number of entries in the array of the correct type.
As a final point of interest, note how the Description method here is passed a hierarchy for the plug-in name. This is an old technique for laying out menus in NUKE. It is less flexible than the newer menu.py python approach, but can be used to insert Ops into pre-existing menu structures. It’s not recommended for third-party development.
Exercise: Build a Mult Node¶
Now, let’s take what we’ve covered so far and apply it. For this first exercise, take the Add.cpp source code in the NDK, get it building, and then:
Change its name to Mult and ensure it can be created in the Node Graph (DAG).
Amend the pixel_engine to multiply, rather than offset, the image data by the user-set value. Use the inbuilt NUKE node ‘Multiply’ as a point of comparison to get this right.
Amend the default value to be 1 rather than 0.
Amend the help and tooltip text to reflect this.
Amend the NukeWrapper call to set the channel knob to be RGB by default, rather than ‘all’.
Relax with a satisfying cup of tea.