GeoReader and GeoWriter: Supporting Custom 3D File Formats¶

Introduction¶

NUKE ships with support for two 3D file formats: OBJ and FBX. It’s possible to extend this to additional formats by writing custom GeoReader and GeoWriter classes.

These are similar to the Reader and Writer classes for image format support. It may be useful to read through that section as well, for additional information.

Mapping File Types to Readers and Writers¶

To identify which reader or writer version is needed, NUKE relies on parsing the file extension and finding a matching reader. If a .my3d extension is encountered, for example, NUKE looks for a reader called my3dReader.<OS specific extension>.

This works well for simple cases, but there’s still a problem when you want one reader to support multiple file extensions - for example if the reader uses a library which can parse a number of different file formats. In this case, you don’t want to have a separate copy of the reader for every different file format it supports. Fortunately, NUKE provides two different systems to deal with this.

Before looking for the reader plugin, NUKE looks for a corresponding TCL plugin, but with .tcl as the extension instead of .dll, .so, or .dylib; for example, my3dReader.tcl when loading a .my3d file. If NUKE finds this TCL file, it executes it to load the correct reader.

So if you have a plugin called MyFancy3DReader.dll which supports reading .my3d files, your my3dReader.tcl file would probably look like this:

# my3dReader.tcl
load MyFancy3DReader

This script is invoked instead of my3dReader.dll and ensures the correctly named MyFancy3DReader.dll is loaded for the read.

The other system works by extending the supported extension types for the reader itself, and is explained below.

These systems apply to writers as well, except that NUKE looks for a plugin or TCL file called, in this case, my3dWriter instead of my3dReader.

The `GeoReader` Base Class¶

class DD::Image::GeoReader¶

This is the base class for things which turn files on disk into 3D for NUKE. You’ll have to subclass this to make your own reader and, to make it do anything useful, you’ll need to override at least one method:

void GeoReader::geometry_engine(Scene &scene, GeometryList &out)¶: Read geometry from a file and populate the GeometryList with it. The name of the file to read from can be obtained with the filename() method.

If your reader needs to do any expensive setup before reading, you can override this too:

void GeoReader::_open()¶: Called once before the first call to geometry_engine, so that you can perform any one-time initialization. The default implementation does nothing.

You’ll also need to provide a GeoReader::Description object as a static member of your subclass. This tells NUKE how to instantiate your reader and what file formats it supports. The description object stores two function pointers: one to a function for creating the reader, and a second for a function that tests whether a file is in a format that the reader understands. You’ll need to provide implementations of these functions too, although the implementation is often quite straightforward.

So the outline of a basic GeoReader implementation looks something like this:

using DD::Image;

class my3dReader : public GeoReader
{
public:
  my3dReader(ReadGeo* readNode);

  virtual void geometry_engine(Scene& scene, GeometryList& out);

  static GeoReader* Build(ReadGeo* readNode, int fd, const unsigned char* buf, int bufSize);
  static bool Test(int fd, const unsigned char* buf, int bufSize);

  static GeoReader::Description description;
};

Implementing the `Build` and `Test` Methods for a GeoReader¶

The Test method is called first to determine whether the data is in a format that our reader understands. At the time of the call, the enclosing ReadGeo2 Op has already read in the first few bytes of the file and passes them in to the Test method. Since most file types start with a few bytes that identify the format, this should be all you need to figure out whether the file is one you can read. You should avoid reading any further data from the file unless you absolutely have to; however if you do have to, the fd parameter contains an open file descriptor that you can read from.

Here’s an example of a Test method for a made-up 3D format, “my3d”, following the example class definition above:

bool my3dReader::Test(int fd, const unsigned char* buf, int bufSize)
{
  const char* kHeader = "{my3d}";
  const int kHeaderLen = strlen(kHeader);

  // Check that the buffer starts with our header.
  return (bufSize >= kHeaderLen && strncmp(kHeader, (const char*)buf, kHeaderLen) == 0);
}

This checks that we have enough bytes in the buffer to contain the file header and the first thing in the buffer is the text “{my3d}”.

If the Test method returns true, the Build method is called to construct the reader. The parameters are identical to the Test method, with an additional one giving a pointer to the enclosing ReadGeo node. This method should construct the reader instance, passing through the node pointer and file descriptor. The buffer can be inspected if you need any further information to construct your reader with; this can be useful when your reader supports several different file types, but in many cases can simply be ignored.

Here’s an example of the Build method for the my3dReader class mentioned above:

GeoReader* my3dReader::Build(ReadGeo* readNode, int fd, const unsigned char* buf, int bufSize)
{
  return new my3dReader(readNode);
}

The constructor for your reader can be as simple or as complex as you like. Here’s what my3dReader looks like:

my3dReader::my3dReader(ReadGeo* readNode) :
  GeoReader(readNode)
{
}

Implementing the `geometry_engine` Method for a GeoReader¶

The geometry_engine method of a GeoReader is where all the interesting stuff happens. It’s responsible for reading the input file and filling in a geometry list with the contents.

You can get the name of the file to read using the filename() method. As you read the file, you’ll be adding objects to the geometry list and setting points, attributes, etc. on those objects. This part is more or less the same as for a SourceGeo::create_geometry implementation, so it might be useful to review the SourceGeo Tutorial alongside this.

Here’s what an implementation for the my3dReader class might look like:

void my3dReader::geometry_engine(Scene& scene, GeometryList& out)
{
  const unsigned int kMaxLineLen = 4096;

  // Open the file
  std::ifstream in(filename());

  // Skip over the header.
  in.ignore(kMaxLineLen, '\n');

  int obj = 0;
  out.add_object(obj);
  PointList* points = out.writable_points(obj);
  Attribute* normals = out.writable_attribute(obj, Group_Primitives, kNormalAttrName, NORMAL_ATTRIB);
  Attribute* texCoords = out.writable_attribute(obj, Group_Points, kUVAttrName, VECTOR4_ATTRIB);

  Vector3 pos, normal;
  Vector4 uv(0, 0, 0, 1);
  float size;
  int n = 0;
  while (in.good()) {
    in >> size >> pos.x >> pos.y >> pos.z >> normal.x >> normal.y >> normal.z >> uv.x >> uv.y;
    in.ignore(kMaxLineLen, '\n'); // skip any trailing characters.

    out.add_primitive(obj, new Triangle(n  * 3, n * 3 + 1, n * 3 + 2));

    Vector3 v1 = normal.cross(Vector3(1, 0, 0));
    Vector3 v2 = v1.cross(normal);
    v1.normalize();
    v2.normalize();
    points->push_back(pos);
    points->push_back(pos + (v1 * size));
    points->push_back(pos + (v2 * size));

    normals->add(1);
    normals->normal(n) = normal;

    texCoords->add(3);
    texCoords->vector4(n * 3) = uv;
    texCoords->vector4(n * 3 + 1) = uv + Vector4(size, 0, 0, 0);
    texCoords->vector4(n * 3 + 2) = uv + Vector4(0, size, 0, 0);

    ++n;
  }
}

This reads a very simplistic made-up file format where each line after the header represents a right-angled triangle as 9 floating point values. The values are, in order:

size: the length for the two shorter sides of the triangle.
x, y, z: The 3D position of the triangle’s right-angle corner.
nx, ny, nz: The direction of the surface normal for the triangle.
u, v: The texture coordinate at the triangle’s right-angle corner (the texture coordinates for the other corners are derived from this).

The first thing we do in this method is open the file for reading and skip past the header:

// Open the file
std::ifstream in(filename());

// Skip over the header.
in.ignore(kMaxLineLen, '\n');

The next step is some object setup. We only ever produce a single object from this reader, so we add it here. We also grab pointers to the points and attribute lists that we add values to while we read from the file. Both writable_points and writable_attribute can be expensive to call, so it’s a good idea to grab these outside of the loop:

int obj = 0;
out.add_object(obj);
PointList* points = out.writable_points(obj);
Attribute* normals = out.writable_attribute(obj, Group_Primitives, kNormalAttrName, NORMAL_ATTRIB);
Attribute* texCoords = out.writable_attribute(obj, Group_Points, kUVAttrName, VECTOR4_ATTRIB);

After that, we declare some variables then get into the main reading loop. The loop will execute once for each line in the file, until we get to the end of the file or there’s some error reading from it:

while (in.good()) {
  in >> size >> pos.x >> pos.y >> pos.z >> normal.x >> normal.y >> normal.z >> uv.x >> uv.y;
  in.ignore(kMaxLineLen, '\n'); // skip any trailing characters.

  // process the data we've just read
}

The processing inside the loop consists of the following steps:

adding a primitive to the current object
doing some calculations to figure out the implicit corner points and texture coordinates
adding the points, surface normal, and texture coordinates to the current object

out.add_primitive(obj, new Triangle(n  * 3, n * 3 + 1, n * 3 + 2));

Vector3 v1 = normal.cross(Vector3(1, 0, 0));
Vector3 v2 = v1.cross(normal);
v1.normalize();
v2.normalize();
points->push_back(pos);
points->push_back(pos + (v1 * size));
points->push_back(pos + (v2 * size));

normals->add(1);
normals->normal(n) = normal;

texCoords->add(3);
texCoords->vector4(n * 3) = uv;
texCoords->vector4(n * 3 + 1) = uv + Vector4(size, 0, 0, 0);
texCoords->vector4(n * 3 + 2) = uv + Vector4(0, size, 0, 0);

The cross products are used to find the vectors along the edges of the triangle: the first one finds an arbitrary vector perpendicular to the surface normal; the second one finds the vector perpendicular to both the first vector and the surface normal. We scale the length of these vectors to be the size we read in earlier and add them to the position from the current line to find the remaining two corners.

The Complete `my3dReader` Code¶

#include "DDImage/Attribute.h"
#include "DDImage/GeometryList.h"
#include "DDImage/GeoReader.h"
#include "DDImage/Triangle.h"
#include "DDImage/Vector3.h"

#include <iostream>
#include <fstream>
#include <string>

using namespace DD::Image;

namespace my3d {

  class my3dReader : public GeoReader
  {
  public:
    my3dReader(ReadGeo* readNode);
    virtual void geometry_engine(Scene& scene, GeometryList& out);

    static GeoReader* Build(ReadGeo* readNode, int fd, const unsigned char* buf, int bufSize);
    static bool Test(int fd, const unsigned char* buf, int bufSize);

    static GeoReader::Description description;
  };


  //
  // my3dReader methods
  //
  
  my3dReader::my3dReader(ReadGeo* readNode) :
    GeoReader(readNode)
  {
  }

  
  void my3dReader::geometry_engine(Scene& scene, GeometryList& out)
  {
    const unsigned int kMaxLineLen = 4096;

    // Open the file
    std::ifstream in(filename());
    
    // Skip over the header.
    in.ignore(kMaxLineLen, '\n');

    int obj = 0;
    out.add_object(obj);
    PointList* points = out.writable_points(obj);
    Attribute* normals = out.writable_attribute(obj, Group_Primitives, kNormalAttrName, NORMAL_ATTRIB);
    Attribute* texCoords = out.writable_attribute(obj, Group_Points, kUVAttrName, VECTOR4_ATTRIB);

    Vector3 pos, normal;
    Vector4 uv(0, 0, 0, 1);
    float size;
    int n = 0;
    while (in.good()) {
      in >> size >> pos.x >> pos.y >> pos.z >> normal.x >> normal.y >> normal.z >> uv.x >> uv.y;
      in.ignore(kMaxLineLen, '\n'); // skip any trailing characters.
      
      out.add_primitive(obj, new Triangle(n  * 3, n * 3 + 1, n * 3 + 2));

      Vector3 v1 = normal.cross(Vector3(1, 0, 0));
      Vector3 v2 = v1.cross(normal);
      v1.normalize();
      v2.normalize();
      points->push_back(pos);
      points->push_back(pos + (v1 * size));
      points->push_back(pos + (v2 * size));

      normals->add(1);
      normals->normal(n) = normal;

      texCoords->add(3);
      texCoords->vector4(n * 3) = uv;
      texCoords->vector4(n * 3 + 1) = uv + Vector4(size, 0, 0, 0);
      texCoords->vector4(n * 3 + 2) = uv + Vector4(0, size, 0, 0);

      ++n;
    }
  }


  //
  // my3dReader static methods
  //

  GeoReader* my3dReader::Build(ReadGeo* readNode, int fd, const unsigned char* buf, int bufSize)
  {
    return new my3dReader(readNode);
  }


  bool my3dReader::Test(int fd, const unsigned char* buf, int bufSize)
  {
    const char* kHeader = "{my3d}";
    const int kHeaderLen = strlen(kHeader);

    // Check that the buffer starts with our header.
    return (bufSize >= kHeaderLen && strncmp(kHeader, (const char*)buf, kHeaderLen) == 0);
  }


  //
  // my3dReader static variables
  //

  GeoReader::Description my3dReader::description("my3d\0", my3dReader::Build, my3dReader::Test);

} // namespace my3d

The `GeoWriter` Base Class¶

class DD::Image::GeoWriter¶

This is the base class for things which turn 3D geometry into files on disk for NUKE. You’ll have to subclass this to make your own writer and, to make it do anything useful, you’ll need to override at least one method:

void GeoWriter::execute(Scene &scene)¶: Write data from a scene to the current file. Use the open() method to open the current file for writing and the close() method when you’re done.

You’ll also need to provide a GeoWriter::Description object as a static member of your subclass. This tells NUKE how to instantiate your writer and what file formats it supports. The description object stores a function pointer to a function for creating the writer. You’ll need to provide an implementation of this too, although the implementation is often quite straightforward.

So the outline of a basic GeoWriter implementation looks something like this:

class my3dWriter : public GeoWriter
{
public:
  my3dWriter(WriteGeo* writeNode);
  virtual void execute(Scene& scene);

  static GeoWriter* Build(WriteGeo* readNode);

  static GeoWriter::Description description;
};

Implementing the `Build` Method for a GeoWriter¶

The Build method is called to construct the reader. The only parameter is a pointer to the enclosing WriteGeo node. This method should construct the writer instance, passing through the node pointer.

Here’s an example of the Build method for the my3dWriter class mentioned above:

GeoWriter* my3dWriter::Build(WriteGeo* writeNode)
{
  return new my3dWriter(writeNode);
}

The constructor for your writer can be as simple or as complex as you like. Here’s what the my3dWriter one looks like:

my3dWriter::my3dWriter(WriteGeo* writeNode) :
  GeoWriter(writeNode)
{
}

Implementing the `execute` Method for a GeoWriter¶

The execute method of a GeoWriter is where all the interesting stuff happens. It’s responsible for writing the scene data to the output file.

You can open the output file for writing using the open() method provided by the base class. This fills in the file member variable with a FILE* that points to the open file. The file member is declared as a void pointer though, so we need to cast it to the right type or the compiler produces lots of warnings.

Here’s what an implementation for the my3dWriter class might look like:

void my3dWriter::execute(Scene& scene)
{
  // If we can't open the file for writing, show an error message and abort.
  if (!open()) {
    geo->critical("my3dWriter: failed to open geometry file for writing");
    return;
  }

  // Write the header.
  FILE* f = (FILE*)file;
  fprintf(f, "{my3d}\n");

  // Loop over all objects and write out a point for the corner of every Triangle primitive.
  GeometryList* objects = scene.object_list();

  for (unsigned int obj = 0; obj < objects->size(); ++obj) {
    GeoInfo& info = objects->object(obj);
    const PointList* points = info.point_list();
    const Attribute* normals = info.get_typed_group_attribute(Group_Primitives, kNormalAttrName, NORMAL_ATTRIB);
    const Attribute* texCoords = info.get_typed_group_attribute(Group_Points, kUVAttrName, VECTOR4_ATTRIB);

    for (unsigned int p = 0; p < info.primitives(); ++p) {
      const Triangle* triangle = dynamic_cast<const Triangle*>(info.primitive(p));
      if (!triangle)
        continue;

      unsigned int corner = triangle->vertex(0);
      Vector3 pos = (*points)[corner];
      Vector3 normal = normals->normal(p);
      Vector4 uv = texCoords->vector4(corner);

      unsigned int corner2 = triangle->vertex(1);
      Vector3 edge = (*points)[corner2] - pos;
      float size = edge.length();

      fprintf(f, "%f %f %f %f %f %f %f %f %f\n", size,
                 pos.x, pos.y, pos.z,
                 normal.x, normal.y, normal.z,
                 uv.x, uv.y);
    }
  }

  close();
}

The code above writes out a brief description of any triangle primitives in the scene. The file format it produces is the same one that the my3dReader class above reads; see the definition above for details.

Note the use of geo->critical to report an error when opening a file. You can use this at other points while writing (or reading) too, to report any errors you encounter.

The first thing we do in this method is open the file for writing, aborting with an error message if we fail for any reason:

if (!open()) {
  geo->critical("my3dWriter: failed to open geometry file for writing");
  return;
}

Next we write the file header:

FILE* f = (FILE*)file;
fprintf(f, "{my3d}\n");

Then we enter the main loop, iterating over all objects in the scene. Inside that loop we iterate over all primitives on each object looking for any Triangle primitives:

GeometryList* objects = scene.object_list();
for (unsigned int obj = 0; obj < objects->size(); ++obj) {
  GeoInfo& info = objects->object(obj);
  const PointList* points = info.point_list();
  const Attribute* normals = info.get_typed_group_attribute(Group_Primitives, kNormalAttrName, NORMAL_ATTRIB);
  const Attribute* texCoords = info.get_typed_group_attribute(Group_Points, kUVAttrName, VECTOR4_ATTRIB);

  for (unsigned int p = 0; p < info.primitives(); ++p) {
    const Triangle* triangle = dynamic_cast<const Triangle*>(info.primitive(p));
    if (!triangle)
      continue;

    // write out the triangle details
  }
}

Note that for each object, we grab pointers to the point list, normals, and texture coords. It’s a bit cheaper to do this outside of the inner loop.

The processing for each Triangle consists of some lookups to get the basic information to write to the value; a calculation to get the value for the size field; and writing the data:

unsigned int corner = triangle->vertex(0);
Vector3 pos = (*points)[corner];
Vector3 normal = normals->normal(p);
Vector4 uv = texCoords->vector4(corner);

unsigned int corner2 = triangle->vertex(1);
Vector3 edge = (*points)[corner2] - pos;
float size = edge.length();

fprintf(f, "%f %f %f %f %f %f %f %f %f\n", size,
           pos.x, pos.y, pos.z,
           normal.x, normal.y, normal.z,
           uv.x, uv.y);

After the main loop, the only thing left to do is close the file. Don’t forget to do this!

The Complete `my3dWriter` Code¶

#include "DDImage/Attribute.h"
#include "DDImage/GeometryList.h"
#include "DDImage/GeoWriter.h"
#include "DDImage/Scene.h"
#include "DDImage/Triangle.h"
#include "DDImage/Vector3.h"
#include "DDImage/Vector4.h"

#include <cstdio>

using namespace DD::Image;

namespace my3d {

  class my3dWriter : public GeoWriter
  {
  public:
    my3dWriter(WriteGeo* writeNode);
    virtual void execute(Scene& scene);

    static GeoWriter* Build(WriteGeo* readNode);

    static GeoWriter::Description description;
  };


  //
  // my3dReader methods
  //
  
  my3dWriter::my3dWriter(WriteGeo* writeNode) :
    GeoWriter(writeNode)
  {
  }

  
  void my3dWriter::execute(Scene& scene)
  {
    // If we can't open the file for writing, show an error message and abort.
    if (!open()) {
      geo->critical("my3dWriter: failed to open geometry file for writing");
      return;
    }

    // Write the header.
    FILE* f = (FILE*)file;
    fprintf(f, "{my3d}\n");

    // Loop over all objects and write out a point for the corner of every Triangle primitive.
    GeometryList* objects = scene.object_list();
    for (unsigned int obj = 0; obj < objects->size(); ++obj) {
      GeoInfo& info = objects->object(obj);
      const PointList* points = info.point_list();
      const Attribute* normals = info.get_typed_group_attribute(Group_Primitives, kNormalAttrName, NORMAL_ATTRIB);
      const Attribute* texCoords = info.get_typed_group_attribute(Group_Points, kUVAttrName, VECTOR4_ATTRIB);

      for (unsigned int p = 0; p < info.primitives(); ++p) {
        const Triangle* triangle = dynamic_cast<const Triangle*>(info.primitive(p));
        if (!triangle)
          continue;
        
        unsigned int corner = triangle->vertex(0);
        Vector3 pos = (*points)[corner];
        Vector3 normal = normals->normal(p);
        Vector4 uv = texCoords->vector4(corner);

        unsigned int corner2 = triangle->vertex(1);
        Vector3 edge = (*points)[corner2] - pos;
        float size = edge.length();

        fprintf(f, "%f %f %f %f %f %f %f %f %f\n", size,
                   pos.x, pos.y, pos.z,
                   normal.x, normal.y, normal.z,
                   uv.x, uv.y);
      }
    }

    close();
  }


  //
  // my3dReader static methods
  //

  GeoWriter* my3dWriter::Build(WriteGeo* writeNode)
  {
    return new my3dWriter(writeNode);
  }


  //
  // my3dReader static variables
  //

  GeoWriter::Description my3dWriter::description("my3d\0", my3dWriter::Build);

} // namespace my3d

GeoReader and GeoWriter: Supporting Custom 3D File Formats¶

Introduction¶

Mapping File Types to Readers and Writers¶

The `GeoReader` Base Class¶

Implementing the `Build` and `Test` Methods for a GeoReader¶

Implementing the `geometry_engine` Method for a GeoReader¶

The Complete `my3dReader` Code¶

The `GeoWriter` Base Class¶

Implementing the `Build` Method for a GeoWriter¶

Implementing the `execute` Method for a GeoWriter¶

The Complete `my3dWriter` Code¶

Table of Contents

Previous topic

Next topic

GeoReader and GeoWriter: Supporting Custom 3D File Formats¶

Introduction¶

Mapping File Types to Readers and Writers¶

The GeoReader Base Class¶

Implementing the Build and Test Methods for a GeoReader¶

Implementing the geometry_engine Method for a GeoReader¶

The Complete my3dReader Code¶

The GeoWriter Base Class¶

Implementing the Build Method for a GeoWriter¶

Implementing the execute Method for a GeoWriter¶

The Complete my3dWriter Code¶

The `GeoReader` Base Class¶

Implementing the `Build` and `Test` Methods for a GeoReader¶

Implementing the `geometry_engine` Method for a GeoReader¶

The Complete `my3dReader` Code¶

The `GeoWriter` Base Class¶

Implementing the `Build` Method for a GeoWriter¶

Implementing the `execute` Method for a GeoWriter¶

The Complete `my3dWriter` Code¶