How to capture frames in a MacOS MTKView for recording to a video I have created a 3D model of a local outdoor performance space and I have an app that uses Metal on MacOS 10.14.2 to display the model. I want to create an animation by flying the camera around the scene, while recording each frame (actually every other frame for 30 fps). I know how to do the animated fly-around and I know how to create a video frame-by-frame with AVFoundation. The step for which I can find no information is how I can capture each frame.

I have a completion handler so I know when the gpu has finished each command buffer.

But what is the best way to get the image in CPU space?

I thought perhaps I could do this by attaching a second texture to colorAttachments[1] but this has resulted in some odd behavior where the original scene that used to fill my MTKView window now occupies just the upper left quadrant of the window.

What I was trying to do is write the same color to both colorAttachments[0] (my screen) and to colorAttachments[1]. In my shader I defined:

struct FragmentOut {
    float4 color0 [[ color(0) ]];
    float4 color1 [[ color(1) ]];
};

My fragment shader looks like:

fragment FragmentOut grove_fragment_function(
    VertexOut vIn [[ stage_in ]],
    constant Uniforms &uniforms [[buffer(1)]]) {
 ....
    float4 color = diffuseColor + ambientColor + specularColor;
    out.color0 = color;
    out.color1 = color;
    return out;
}

My hope was that I could then use something like:

offScreenTextureBuffer?.getBytes(buffer, bytesPerRow: 4 * w, from: MTLRegionMake2D(0, 0, w, h), mipmapLevel: 0)

to transfer the image data to a local buffer.
This doesn't seem to work, plus I have the unexpected display behavior noted above.

I am configuring the offScreenTextureBuffer thus:

let pixelFormat = MTLPixelFormat.bgra8Unorm_srgb
var offScreenBufferDescriptor =  MTLTextureDescriptor.texture2DDescriptor(pixelFormat: pixelFormat,                                                                             width: 1000,                                                                             height: 600,                                                                            mipmapped: false)
offScreenBufferDescriptor.usage = [.renderTarget, .shaderRead]
offScreenTextureBuffer = device.makeTexture(descriptor: offScreenBufferDescriptor)

I also tried to use the presentation layer from my MTKView. I can get the texture from the presentation layer but when I try to use the getBytes() func on the returned texture I get an error because the buffer is configured as "framebufferonly". A presentation layer that can't be read doesn't seem particularly useful.

Any suggestions on how to capture each frame would be appreciated.

I've seen no answer to my question. But after lots of experimentation and further study of Apple documentation for clues I have figured out how to capture GPU generated frames.

First, I gave up on the attempt to have the shader write to the screen and an offscreen buffer in the same render pass. The odd (and undocumented as far as I can tell) behavior of MTKView in which the onscreen view occupies only a portion of the original area of the screen. I think this a bug in MTKView. It seems to act as if each attachment should appear somewhere on the screen. MTKView is not checking to see if the attachments are all on the screen.

Since MTKView calls the draw() function 60 times per second I decided to alternate. One draw() goes to the screen and the following draw() goes to my off screen buffer. This gives me a 30 Hz frame rate for the captured video frames, which is what I wanted anyway. I can watch the screen and see what will appear in the video.

For the video capture draw() calls I now define the MTLTextureDescriptor in this way:

let offScreenPixelFormat = MTLPixelFormat.rgba32Float   // 32Float is convenient for AVFoundation video recording
let offScreenBufferDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: offScreenPixelFormat,
          width: offW, height: offH, mipmapped: false)
offScreenBufferDescriptor.usage = [.shaderWrite, .renderTarget]
offScreenBufferDescriptor.storageMode = .managed
offScreenBufferDescriptor.resourceOptions = [.storageModeManaged]
offScreenBufferDescriptor.textureType = MTLTextureType.type2D
offScreenTextureBuffer = device.makeTexture(descriptor:    offScreenBufferDescriptor)

It turns out this still did not work. I got nothing back from the GPU. That is, I got nothing back until I added this code to the end of the commandBuffer:

  let blitCommandEncoder = commandBuffer.makeBlitCommandEncoder()
  blitCommandEncoder?.synchronize(resource: offScreenTextureBuffer!)
  blitCommandEncoder?.endEncoding()

This causes the GPU memory version of the offScreenTextureBuffer to be copied back to the CPU memory version. I figured this out by looking at the Apple documentation. Like all Apple documentation, it doesn't tell you why you might want to synchronize. But you definitely do need to do this. I have seen several online code examples where this is NOT done and they claim to get data back from the GPU.
I tried one example code without the synchronize command and nothing was returned. So I assume the requirement to synchronize is a new feature and the online code is just out of date. It would be nice if the Apple documentation made it clear this is now a requirement.

With these changes I can retrieve GPU frames. I have created a short video using GPU-generated frames.

If anyone is interested in seeing a complete code sample let me know.


How to handle Half edge boundary edge iteration from vertex? Ok so let's say we have a portion of a mesh that looks somewhat like this:

A triangle fan with one of the triangles removed. If that triangles was not removed, we could get all the edges adjacent to that vertex this way:

std::vector<GraphEdgePtr> GraphVertex::GetVertexEdges()
{
    std::vector<GraphEdgePtr> edges;
    auto current_edge = edge;
    do
    {
        edges.push_back(current_edge);
        current_edge = current_edge->pair->next;
    } while(current_edge != edges[0]);

    return edges;
}

However, if the triangle is removed, the above will segfault. One could create an additional loop that iterates backwards from the starting edge in the case the first loop finds a null pointer.

However that is a bit inelegant in my opinion. Is there a way to do it in a single loop? Using a stack or a queue or some sort for example?

It depends a little on your actual data structure and how it even encodes boundary edges. What I find more useful than representing the boundary with a null half edge pointer, and thus having this situation of a half edge without an opposite and thus a hard stop for any such iteration, is instead encoding the boundary with a full half edge pair too, but with one of the half edges having a null face pointer.

So the boundary edge actually does have an opposite (or pair pointer) and that opposite is fully integrated into the pointer structure, complete with next pointer and everything. Just that its face pointer in null, thus denoting an empty face or hole. So basically every hole is represented by a null face with half edges going around it.

In addition to being able to iterate around vertices regardless of the starting point, this also has the advantage that you can easily iterate around boundary curves. And you can even have more than one hole at a vertex (although, that can still be problematic in other situations). Of course you might still have to handle the boundary half edges specially in some situations, especially when you're using the face pointers, but you won't get around that anyway if your mesh can have holes.

If that is too much of a hassle to restructure your data and algorithms this way, though, maybe you can instead adjust your vertex starting edge pointers to always point to the boundary edge if there is one. You'd of course have to make sure that stays consistent, but if you can assume that as a precondition, you can just start iterating from there. And you can also check immediately if the vertex is at a boundary without having to iterate.


What is required to limit the amount of draw calls in Unity? I would like to limit the amount draw calls to approximate performance. Is there a nominal amount to aim for when working with high performance gaming?

From my understanding to render an object that contains exceptional amount of vertices, this would require a lot of draw calls. Though, without doing dynamic draw calls, is there another way to limit the amount of draw calls?

Per example, if the number draw calls is to be 60 for 60 frames per second; this rate is completely arbitrary, am just looking for a basic explanation on how to control this.

There's quite a bit of information about this:

Static/dynamic batching: basically combining objects into one super-object
Atlas textures: combining textures into one large texture than only uses one material
Switch to a deferred renderer if you are using many lights

  From my understanding to render an object that contains exceptional amount of vertices, this would require a lot of draw calls.

The vertex amount doesn't directly determine the number of draw calls. There also isn't really a guide on how many draw calls is too much. The best thing to do is to profile your application and see where your performance problems come from.


How are mipmap levels computed in Metal? My question is specifically in regards to Metal, since I don't know if the answer would change for another API.

What I believe I understand so far is this:

A mipmapped texture has precomputed "levels of detail", where lower levels of detail are created by downsampling the original texture in some meaningful way.
Mipmap levels are referred to in descending level of detail, where level 0 is the original texture, and higher-levels are power-of-two reductions of it.
Most GPUs implement trilinear filtering, which picks two neighboring mipmap levels for each sample, samples from each level using bilinear filtering, and then linearly blends those samples.

What I don't quite understand is how these mipmap levels are selected. In the documentation for the Metal standard library, I see that samples can be taken, with or without specifying an instance of a lod_options type. I would assume that this argument changes how the mipmap levels are selected, and there are apparently three kinds of lod_options for 2D textures:

bias(float value)
level(float lod)
gradient2d(float2 dPdx, float2 dPdy)

Unfortunately, the documentation doesn't bother explaining what any of these options do. I can guess that bias() biases some automatically chosen level of detail, but then what does the bias value mean? What scale does it operate on? Similarly, how is the lod of level() translated into discrete mipmap levels? And, operating under the assumption that gradient2d() uses the gradient of the texture coordinate, how does it use that gradient to select the mipmap level?

More importantly, if I omit the lod_options, how are the mipmap levels selected then? Does this differ depending on the type of function being executed?

And, if the default no-lod-options-specified operation of the sample() function is to do something like gradient2D() (at least in a fragment shader), does it utilize simple screen-space derivatives, or does it work directly with rasterizer and interpolated texture coordinates to calculate a precise gradient?

And finally, how consistent is any of this behavior from device to device? An old article (old as in DirectX 9) I read referred to complex device-specific mipmap selection, but I don't know if mipmap selection is better-defined on newer architectures.

Mip selection is pretty well standardized across devices today—with the exception of some of the nitty-gritty details of anisotropic filtering, which is still up to the individual GPU manufacturers to define (and its precise details are generally not publicly documented).

A good place to read about mip selection  in detail is in the OpenGL spec, section 8.14, "Texture Minification". I'd assume it works the same way in Metal. (Apple could have changed something, considering they make both the hardware and the API...but I doubt they have.) I'll summarize it here.

The default mip selection (not using any of the lod_options modifiers) uses the screen-space gradients of the texture coordinates to pick the mip levels. Essentially, it tries to pick the mip levels that produce as close as possible to a 1:1 mapping of texels to pixels. For example, if the gradients have a length of 4 texels per pixel, it would pick mip level 2 (which is 1/4th the size of level 0 and therefore gives you 1 mipped texel per pixel).

With trilinear filtering, since you usually don't land on an exact 1:1 mapping, it picks the two nearest levels and linearly interpolates between them, so that you have a smooth transition between mip levels as the camera or objects in your scene move around.

To be mathematically precise: you take the screen-space gradients of the texture coordinates, scale them by the texture size (to get them in units of "texels per pixel"), take their lengths, take the longer of the two (X and Y) gradients, and calculate its logarithm base 2. This quantity is called $\lambda$ in the OpenGL spec; it's also commonly known as the texture LOD (level of detail). It's simply a continuous version of the integer mip level index. So, the mip levels used are the two integers nearest to $\lambda$, and the fractional part of $\lambda$ is used to blend between them. For example if $\lambda = 2.8$, the GPU will sample mip levels 2 and 3 of the texture, then blend to 80% level 3 and 20% level 2.

If anisotropic filtering is turned on, then instead of simply using the longer of the two gradients, you use the ratio of them to set the number of aniso samples. For example, if the X gradient is 4 times longer than the Y gradient, you'd use 4 aniso samples, with their positions spaced out along the X gradient. Each one would be a trilinear sample, using a $\lambda$ corresponding to 1/4th the length of the X gradient (i.e. two mip levels lower, since $\log_2(1/4) = -2$).

Now, as for the modifier options:

bias applies an offset to $\lambda$ before using it to select mip levels. So, for example, a bias of +1 will make it do trilinear filtering using mips one level higher than it would normally. A bias of −1 will use mips one level lower, and so on.
level completely overrides the automatic calculation of $\lambda$ from screen-space gradients, and lets you put in your own value of $\lambda$ (aka lod) directly.
gradient2d lets you put in your own gradient vectors, which substitute for the implicit screen-space gradients of the texture coordinates. The rest of the mip selection and sampling process proceeds as normal, but with the altered gradients. This lets you customize anisotropic filtering.

In other types of shaders besides fragment shaders, there's no notion of "screen-space gradients", so the latter two operations are typically the only ones allowed—any operation that tries to use implicit gradients would give a compile error. I'm not positive that's how Metal does it, but that's what I'd expect from working with other APIs.


A question about derivation in oblique view frustum Problem Description:

Reference:
Oblique View Frustum Depth Projection and Clipping

"Mathematics for 3D Game Programming and Computer Graphics" 3rd edition, chapter 5.6.
M is the original projection matrix.
Plane C is the modified near plane, and C'=transpose(inverse(M))*C is the projected plane by M.
Why in NDC the corner Q' of the view frustum lying opposite the plane C' is given by Q'=(Cx, Cy, 1, 1)?

check this link

quotations ：

“The points (+/-1, +/-1, 1, 1) are the four corners of the view frustum on the far plane in clip space.”

this should be the point Q' which equals (sgn(Cx), sgn(Cy), 1, 1)


How to use textures with DSA? My question is very similar to this one but the answers there did not help me.

I am trying to use the DSA approach to bind textures and it is not working.

Here is my attempt at the DSA code, which leads too all black in as my output:

GLuint samplerName;
glCreateSamplers(1, &samplerName);
glSamplerParameteri(samplerName, GL_TEXTURE_WRAP_S, GL_REPEAT);
glSamplerParameteri(samplerName, GL_TEXTURE_WRAP_T, GL_REPEAT);
glSamplerParameteri(samplerName, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glSamplerParameteri(samplerName, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glBindSampler(0, samplerName);

// Create texture
GLuint texture;
glCreateTextures(GL_TEXTURE_2D, 1, &texture);
glTextureStorage2D(texture, 1, GL_RGB, width, height);
glTextureSubImage2D(texture, 0, 0, 0, width, height, GL_RGB, GL_UNSIGNED_BYTE, data);
glTextureParameteri(texture, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glBindTextureUnit(0, texture);

If I add these two lines from the older approach:

glBindTexture(GL_TEXTURE_2D, texture);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, width, height, 0, GL_RGB, GL_UNSIGNED_BYTE, data);

it works so I do not think it is an issue with my sampler or my data. Does anybody have any advice here?

My issue was in the call to glTextureStorage2D(texture, 1, GL_RGB, width, height);

From the man page

  internalformat must be one of the sized internal formats given in Table 1 below, one of the sized depth-component formats GL_DEPTH_COMPONENT32F, GL_DEPTH_COMPONENT24, or GL_DEPTH_COMPONENT16, one of the combined depth-stencil formats, GL_DEPTH32F_STENCIL8, or GL_DEPTH24_STENCIL8, or the stencil-only format, GL_STENCIL_INDEX8

Changing GL_RGB to GL_RGB8 fixed my issue.


Background behind 3D model I have object loader, which i want to have background. I tried to put jpg image like texture to my application. I tried to make that with FreeImage library, but the image didn't show. Also i tried with Gdiplus, i opened the image but when i opened and the object the background disappeared.
I send link with the code (with FreeImage library), If someone have any ideas i will be so happy, because I haven't yet.

https://github.com/Nushiii/viewer/blob/master/ViewerPicture%20FreeImege/ViewerPicture/main.cpp

EDIT:

       int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nShowCmd)
    {

    MSG msg = {0};
    WNDCLASSEX wcl = {0};

    GLuint texture;
    float theta = 0.0f;

    wcl.cbSize = sizeof(wcl);
    wcl.style = CS_OWNDC | CS_HREDRAW | CS_VREDRAW;
    wcl.lpfnWndProc = WindowProc;
    wcl.cbClsExtra = 0;
    wcl.cbWndExtra = 0;
    wcl.hInstance = g_hInstance = hInstance;
    wcl.hIcon = static_cast<HICON>(LoadImage(hInstance,
    MAKEINTRESOURCE (IDI_ICON1),
    IMAGE_ICON,
    48, 48,    // or whatever size icon you want to load
    LR_DEFAULTCOLOR));
    wcl.hCursor = LoadCursor(0, IDC_ARROW);
    wcl.hbrBackground = 0;
    wcl.lpszMenuName = MAKEINTRESOURCE(MENU_FIXED_FUNC);
    wcl.lpszClassName = "GLWindowClass";
    wcl.hIconSm = 0;

    if (!RegisterClassEx(&wcl))
        return 0;

    g_hWnd = CreateAppWindow(wcl, APP_TITLE);

          // load our texture
      FIBITMAP  *dib1 = loadImage("angel.jpg");
      texture = LoadTexturee( dib1, TRUE );

    if (g_hWnd)
    {
        SetProcessorAffinity();

        if (Init())
        {
            ShowWindow(g_hWnd, nShowCmd);
            UpdateWindow(g_hWnd);

            while (true)
            {
                if (PeekMessage(&msg, 0, 0, 0, PM_REMOVE))
                {
                    if (msg.message == WM_QUIT)
                        break;

                    TranslateMessage(&msg);
                    DispatchMessage(&msg);
                } else {

EDIT

                  UpdateFrame(GetElapsedTimeInSeconds());
                  DrawFrame(texture, g_hDC);
                  SwapBuffers(g_hDC);

                }

                if (msg.message == WM_QUIT)
                    break;

            }
        }

        Cleanup();
        UnregisterClass(wcl.lpszClassName, hInstance);
    }
    FreeTexture( texture );
    return static_cast<int>(msg.wParam);
}

EDIT: Now i load the picture, but the object doesn'n load.

VOID DrawFrame(GLuint texture,HDC hDC)
{

glMatrixMode(GL_PROJECTION);
glPushMatrix();
glLoadIdentity();

glMatrixMode(GL_MODELVIEW);
glPushMatrix();
glLoadIdentity();

  // setup texture mapping
 glEnable( GL_TEXTURE_2D );
 glBindTexture( GL_TEXTURE_2D, texture );

 glPushMatrix();
 glBegin( GL_QUADS );
 glTexCoord2d(0.0,0.0); glVertex2d(-1.0,-1.0);
 glTexCoord2d(1.0,0.0); glVertex2d(+1.0,-1.0);
 glTexCoord2d(1.0,1.0); glVertex2d(+1.0,+1.0);
 glTexCoord2d(0.0,1.0); glVertex2d(-1.0,+1.0);
 glEnd();
 glPopMatrix();

glClear(GL_DEPTH_BUFFER_BIT);
glMatrixMode(GL_PROJECTION);
glPopMatrix(); // Pops our orthographic projection matrix
glMatrixMode(GL_MODELVIEW);
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
gluPerspective(CAMERA_FOVY,
   static_cast<float>(g_windowWidth) / static_cast<float> (g_windowHeight),
   CAMERA_ZNEAR, CAMERA_ZFAR);
    glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
gluLookAt(g_cameraPos[0], g_cameraPos[1], g_cameraPos[2],
  g_targetPos[0], g_targetPos[1], g_targetPos[2],
  0.0f, 1.0f, 0.0f);

}

Normally if you paint a background image you simply draw a textured full-screen square while depth testing and writing is disabled before drawing the actual scene (where you re-enable depth)

In the code you posted you draw the quad but then clear the screen again. Don't do that. Move the drawing of the quad from main to right after the glClear call.


library / tool for implementing algorithms I'm new to graphics and want to implement algorithms like dda,scan-line, clipping,etc. I'm confused which library to use. OpenGl, graphics in turbo c,or any other. Please suggest.

If you're completely new to graphics programming I would start with Processing. It's based on Java and JOGL but comes with a whole bunch of drawing commands built in, and its own basic IDE. It's very good for prototyping algorithms and getting quick visual feedback without having to write a ton of boilerplate code, and it insulates you from the underlying API (OpenGL in this case). Presumably you're not going for high performance, since the algorithms you mention are efficiently implemented in hardware, and you just want to familiarize yourself with how they work? If so it's a great place to start.

Once you've got the hang of Processing you can move over to OpenFrameworks, which is very similar but in C++, with the option to 'drill down' and call the underlying OpenGL commands directly. From there, the sky's the limit. :-)


How to calculate position from depth pass of the ZED camera? I'm using the ZED 2 camera, and although the API provides a means of getting a point cloud position from a specific pixel, for my project I need to be able to perform this calculation myself from the depth pass alone. According to the documentation:
"Depth maps captured by the ZED store a distance value (Z) for each pixel (X, Y) in the image. The distance is expressed in metric units (meters for example) and calculated from the back of the left eye of the camera to the scene object."
So my question is, given a pixel index (therefore UV value) and a depth value, how can I work out the world location of the pixel/point? I suspect this has something to do with inverting a matrix (camera? projection?), and I have the following calibration values for the left sensor of the camera:
fx: 1057.1   fy: 1056.71
cx: 979.01   cy: 531.934
k1: -0.0412   k2: 0.0095   k3: -0.0047
p1: -0.0005   p2: -0.0002

But matrices are far from my strong suit so I'm struggling to understand how to turn the above information into a matrix which I could use to extrapolate the position. Can anyone help me?

zed2-camera-datasheet.pdf tells the

Baseline is 120mm
Focal length is 2.12mm

such that based on the fx,fy, you can find the pixel to mm ratio.
You have the local length, baseline, the object coordinates, the principle coordinates. The triangulation is formed, and the depth estimation can be found using ZED depth formula


Why use a tent filter in path tracing? The 99 lines of C path tracer Smallpt renders a 2x2 subpixel grid for each pixel it intends to render and then does a tent filter to combine them.

There is an interesting presentation explaining the code here, and it mentions the tent filter but doesn't explain why it's there.

Can anyone explain why a tent filter would be preferable in this case over a box blur (just averaging the samples)?

Would it be higher quality to go with something better than a tent filter, such as bicubic hermite interpolation?

The theoretical ideal antialiasing filter for discretely sampled data is a sinc filter, because it perfectly removes all frequencies higher than the Nyquist frequency, while leaving alone all the lower ones. So, to some extent, we can expect antialiasing filters that more closely resemble the sinc filter to produce better-quality images.

The tent filter (triangle filter) certainly resembles the central peak of the sinc filter more closely than does the box filter:

A bicubic filter (e.g. Mitchell-Netravali) could capture the shape of the sinc even more precisely, including its first two negative lobes.

The reality of filter selection is a bit more subtle than "approximate sinc as well as possible", since there are different kinds of artifacts that can be generated by "non-ideal" antialiasing filters, such as aliasing, overblurring, and ringing. Also, different filters may be more or less computationally expensive. So it's a game of trying to trade off the different artifacts against each other and against performance. Different scenes/images may favor one choice or another, and it's also partly an aesthetic judgement.

As for why smallpt uses a tent filter in particular, I would guess for a combination of performance (it's a quick filter to evaluate) and brevity—it can be done in a couple lines of code, while a bicubic filter would take a bunch more code.

Incidentally, smallpt actually uses a 2x2 subpixel grid and places a tent filter at each subpixel, then averages together the results of the four subpixels. So the overall effect is, curiously, that of the sum of four tents, which ends up looking like a pyramid with a flat top:

I'm not sure if this was intentional, or just happened to be the way it worked out. My guess is this results in a somewhat sharper image than if a single tent filter per pixel were used (because of the narrower support), but probably also more visible aliasing.


How is VR different from a monitor Apparently Macintosh computers cannot handle the Oculus Rift, because of their 'inferior' graphics cards. But should VR not just be like an external monitor? And concerning computer graphics, how are VR games any different from other games the Macintosh can run easily?
Where is the problem?

VR requires a much higher resolution and framerate, and a much lower latency (start of draw to showing on screen).

If any of those things are missing then people tend to get nauseated.

All of those require much higher bandwidth and compute power than Apple's cards are supposed to be able to handle.


Computing perspective directly Assume the screen is axis aligned with origin at 0 and there is absolutely no view transformation.

in other words we will simply try to project the model onto the screen  plane following perspective. Say our FOV is 60 degrees, so the focal length is tan(30).

I am given a vertex $V=(x,y,z)$

is the projected vertex $V_p = V/(z*tan(30))$ or $V_p = V/(z/tan(30))$

Copying this from another thread where i posted this as the answer but as Wyck suggested, the correct answer is the first one.

There is the whole derivation of it but I'll be discussing a brief overview.
This is for the perspective projection  where the line joining the eye and the center of the projection/image plane is perpendicular to it. Like here

As we can easily see, by similar triangles $\triangle ABC$ and $\triangle AEF$ we have

$Y_p / Y = D/-Z$

where $Y_p$ is the projected $Y$ coordinate on to the image plane. $AB = D$ is the total distance from eye to the image plane and is usually set to 1.

Hence we have,

$Y_p = (Y*D)/-Z$

We  can easily ee from here that,

$tan(\theta_{fov}/2) = 1/AB$

$tan(\theta_{fov}/2) = 1/D$

$D = 1 /tan(\theta_{fov}/2)$

Hence we have,
$Y_p = \frac{Y}{-Z*tan(\theta_{fov}/2)}$

For more details check the complete answer here


What are the fConstX in Spherical Harmonics? At page (slide in reality) 12/20 of this tutorial it is shown a function used to project lights into SH coefficients, however the value of those coefficients is not given, what are their values and what they are for?

Do I have to clamp colors to max value of 1 (float) / 255(uint) when adding lights to SH coefficients?

I'm not an expert on Spherical Harmonic (SH), but I'll try to answer.

SH functions are basis functions. That means that they encode a function $f$ this way by a given basis function $B_i$:

$$\int_{}^{} f(x) B_i(x) dx=fConst_i$$

So what we can do is to use the $fConst_i$ to reconstruct an approximation of the function $f$ this way:

$$\sum_{n=1}^{i} fConst_iB(i)$$

In your case they could look like this:

$$
fConst1 = 0.429043;
$$
$$
fConst2 = 0.511664;
$$
$$
fConst3 = 0.743125;
$$
$$
fConst4 = 0.886227;
$$
$$
fConst5 = 0.247708;
$$

These coefficients approximate diffuse lighting.


How does directional Gaussian blurring work? In image editing software, you often have the ability to do a directional blur where you can choose and angle and a magnitude of a blur.

How does this actually work in practice with Gaussian blurring?

I could see possibly doing something like making the blur go across the axis specified by the angle and use some sort of texture sampling to find the appropriate sample (bilinear or bicubic sampling), but that seems like it probably isn't correct.

How does directional blurring usually work?

I would say this is in fact motion blur, but I'm asking specifically about the blurring portion, not the whole set up for motion blur within a renderer.

For instance before and after:

If I understand your question, you are asking how to actually perform said directional blur in code? A Gaussian blur is typically done by sampling your image in all directions around your current point (or if in 2 passes, one vertical and one horizontal which equates to the same thing), with a specific set of weights for the falloff.

For a directional blur you just need to change the shape of your kernel to "not be round", for lack of better words.

Say you wanted to smear the image to the left only, you could just accumulate samples (with whatever weights you want) with

currentPixelUV.xy + float2( -1.f * someOffsetScale, 0.f) * invScreenRes.xy * currentIteration;

Bilinear texture filtering will be required for this to work when your resulting uvs are not hitting pixel centers.

The magnitude of the blur is controlled by how far you sample from your original pixel.


Should animations be locked to FPS or Fixed time I have been working on a 3D engine and I am now implementing animations, but I can't find out whether or not to use the FPS of the engine or a fixed time method for the transformation's speed. Could someone please tell me the standard for animation timing. Thanks in advanced.

You should use a fixed time for this. Consider the following: Your computer is super fast, you have high end hardware. Now you animate via FPS and make your animation look great. Let's say you had 120 fps.

Since you did great work, you want to show me. I have bad hardware, everything takes ages. I have to work with just 30 fps - sufficient to call it real-time, but not too nice. It will take four times to completely play your animation on my computer, because you made it follow the fps. Therefore it doesn't convince me at all.

This is even more problematic, if you do this in a multiplayer scenario. You could already start a new action after the animation, whereas I have to wait for the animation to finish. Therefore, there is an inconsistency in game state as well as an unfair game, because you can do things much faster.


Collision detection methods for 2D and 3D environments I found many sites, books and articles, all about especifics algorithms, I listed some of them (like bounding boxes, bounding spheres, octrees and KD-tree), but any of these founds lists every algorithm for collision detection, and thats what I need.

Someone knows the name of all algorithms (or at least some of them, any other algorithm would help) for collision detection, or knows where I can find it?

There is a book about real-time collision detection that contains a variety of algorithms for all sorts of intersection tests and lists a large number of collision detection algorithms. It's a decent overview if that's what you are looking for:

http://realtimecollisiondetection.net

https://books.google.com/books?isbn=0080474144


Procedural generation of biological models I was wondering if anyone can point me in the direction of some research papers, or articles that describe a method for generating biological 3D models.
I am drafting a prototype for a new game that would feature a near infinite number of creatures.
I have laid out 2 possible types of procedural generation, the first is more complicated but more desirable.
1st idea
Each of these creatures should be entirely generated from a set of rules, for example:

Locomotion type = legged, limbless, rolling
Number of legs
Number of arms

2nd idea
A base template is used with parameters that modify the overal appearance of the final creature
I realise its an incredibly complicated topic, I've also realised theres a severe lack of resources on the subject - it seems to be extremely cutting edge.
For reference, they achieve something similar in this demo of No Man's Sky https://youtu.be/K0umGtw90Z4?t=96 (thats if its not just staged)
I found the following paper:
https://link.springer.com/article/10.1007/s00371-018-1589-4 but am yet to pay the subscription fee in order to access it - if anyone has read it, I would love to hear if you think it would help me out.
And this also seems like a valuable resource https://nccastaff.bournemouth.ac.uk/jmacey/MastersProjects/MSc13/06/Jon%20Hudson%20Thesis.pdf
If anyone has any experience with this subject , I would love to pick your brain.
Thanks!

I am playing with L-Systems and with some thorough understanding of how they work I think you could manage to get something useful out of them. You can see Paul Bourke's site for a sneak peek of what they do and you can read the  book The Algorithmic Beauty of Plants by Przemyslaw Prusinkiewicz and Aristid Lindenmayer. I am going to try and explain what L-Systems do in case the links die, L-Systems are a way of creating simple rules that define the topology of plants/organisms/fractals by simple axioms, this axioms can be rewritten in terms of other rules that can create incredibly organic behaviours, originally this systems were abstract in the sense that they lacked geometric meaning, but over the years people have figured out how to transform this topologies into geometry through interpreting the rules as "directions" much like turtle graphics. I have seen this systems replace basic geometric rules such as "draw a line between two points" into create a complex building with "A or C characteristics" so I think you could define your interpreter in terms of biologically accurate creatures.


How can I draw a tube on basis of position and angle between two 3d points? I'm struggling with this idea of creating two particles in 3d and draw a tube between them. I can't figure out how to properly set the angle of the tube. I've used two approaches:

calculating the angle with the square root dot product divided by the length of the vectors.
cartesian to polar function which calculates the y and z rotation values

I'm obviously missing something (and I'm not that good in math). So what's my approach?

Thanks in advance! See below what I have so far.

The simplest solution is to use a lookat matrix. This is a matrix calculated from a "eye" point, a "target" point and an up vector.

The resulting matrix will then make the Z axis point from the eye to the target and the y axis in the general direction of up.


How can I implement GIMP's "Colorize" function? I have an RGB pixel that I'd like to colorize in my program (I'm using GLSL) very similar to how the GIMP "Colors > Colorize" function works.

In my program, I can easily convert between RGB and HSV. I can therefore get:

pix_h, pix_s, pix_v

And I also define the inputs of the colorize function as:

col_h, col_s, col_v

What calculations can I perform to determine the output pixel hsv?

I think I can get:

out_h = col_h

But I'm a little lost on how to calculate output saturation and value.

EDIT:
Where lum is the luminance of the input pixel:

out_s = clamp( (1 - pix_v) * 2, 0, 1)

out_v = clamp( pix_v * 2, 0, 1)

seems to produce good results when colorize inputs S=1 ([0, 1]) and V=.5 ([0, 1]). I'm still struggling to generalize these formulas for when S!=1 and V!=.5.

It appears to me that in the image, the saturation of the original image is 0 since it is grayscale. (Unless I've misunderstood the question.) When the saturation is 0, the hue is undefined, but it appears that the GIMP is setting the hue to 0. It is then using the sliders from the dialog to modify the HSL. It adds 180° to the hue (so from 0 to 180 which gets you from red to cyan). The saturation is increased by 75 units, and the lightness is kept the same. It may have special protections for black and white, given that the white part of the letter "A" remains white in the final image (although even there, it appears to have an RGB value of (0.99, 1.0, 1.0), whereas on the right, it's fully white (1.0, 1.0, 1.0)).

You can always look at the source code since it's an open source package.


How to capture rendered frames from an application and stream over network I'd like to know (if it is possible), how I would go about getting at the rendered frames from an application.

For example, say I am using some 3D modelling tool (eg Blender, 3DS Max, Maya) and I have my model (eg a coffee mug that rotates constantly). Animation rendering has completed remotely (perhaps on AWS EC2) and so this coffee mug is rotating...

If I were to log into the remote server using some remote desktop application, I'd see the cup rotating, however, I'd like to rather grab the output from that application (the frames) and stream it to my laptop (or another client).

I know of NVidia's AppStream, but I'd like to know how to get at that rendered frames on the remote server and pipe them to a client, without using some remote desktop software.

I realise my question may seem vague (possibly even not making sense), but I'm very new to this and so any help or pointers would be great.

Thanks

I think you're confused on how rendering and animation work.

A traditional model is nothing more than a bunch of triangles. So a model file coming from say Blender, etc. is just a list of vertices for the triangles. (with some added stuff if you want)

Rendering takes the scene definition and transforms a 3D scene (aka the models oriented somewhere in space) and creates a 2D picture.

Animation creates the illusion of movement by showing static pictures very fast (24 frames per second or more).

Therefore, to have a spinning mug, your application would render each frame, then present them, at, say 30 fps.

Now, to stream the data, you can take two paths:

Render each frame on the server and stream them to the client with something like H.264
Stream the graphics commands to the client, and let them render them.

The first option is what something like OBS or Fraps and (maybe) Nvidia GameStream is doing.
There's a paper that covers the 2nd option here


Any good sources on advaced/special rasterization algorithms and techniques and theory behind them? I spent quite some time looking for books on rasterization of vector graphics and related challenges. The best I found is "Raster Tragedy in Low Resolution",which is old and focused on fonts. There are assorted papers on new renderer here and there, but nothing comprehensive.

Is ththere something good I missed?

One rasterization method I like is Haar wavelet rasterization:

http://faculty.cs.tamu.edu/schaefer/research/wavelet_rasterization.pdf

It can render polygons and Bezier curve shape and good for different resolution. I not see this algorithm in computer graphics books.


Why are normal maps blue and purple in color? I am learning about normal mapping. I understood that RGB values are converted into XYZ, but my question is how is it converted and why is the normal map blue and purple in color?

Only tangent space normal maps are primarily blue. This is because the colour blue represents a normal of (0,0,1) which would be an unchanged normal when the triangle lies in the plane x and y, i.e. perpendicular to the surface. The tangent, x and bi-tangent, y (also referred to as bi-normal) are encoded in the red and green channels and these form to create the a tangent space normal for a point on the surface of the triangle.

If a tangent space normal map was to encode a colour only in red (1.0, 0.0, 0.0) this would generate a tangent-space normal parallel to the triangle surface. This is never seen because it would mean that the triangle would only ever be lit at 90degrees from the surface and view vector at which point you wouldn't be able to see the triangle anyway.

World space normal maps encode the unit normal over a sphere so can be primarily different colours once encoded from [-1, 1] to [0, 1] per channel.

A comparison can be seen here:

In practise normal maps are usually encoded in a 2 channel format such as BC5 which actually only stores the x and y with the z being reconstructed as we know it's a unit vector. This allows you to maintain higher precision with more bits without increasing the file size.


What kind of technology would be involved in the rendering of human nails? There are many references related to the physically-based rendering of several natural features of the numan body, such as skin, hair, eyes.
However, I could not find specific information about the simulation of visually realistic nails. For example, I did not find specificities of the characteristics of the reaction of human nails to light.

A few ideas I had:

According to the Wikipedia article, nails are "made of a tough protective protein called keratin. (...) It is made up of dead skin cells".
Hair is also composed of keratin, so nail rendering might have a few similarities with hair rendering? However I think the main feature of hair rendering is the coloring of specular highlights due to internal reflections inside the hair fiber. Is that phenomenon significant or does it even happen for nails?
Examination indicates that nails are transparent and that they host sub-surface scattering effects. Are there models combining these phenomenons?
The proximity to skin also seems to suggest that the skin under nails has singular appearance properties. Should this layer require some specific techniques to be simulated too?

What kind of technology would be involved to simulate accurately the rendering of human nails?
What kind of tricks digital artists use to simulate the appearance of human nails?

I'm not aware of any rendering technology designed specially for fingernails. Just eyeballing it, I would suggest that a combination of subsurface scattering with a relatively smooth glossy specular surface would get you most of the way there. In other words, you could use the same shader as you do for skin, but with different textures and different specular parameters.

I wouldn't guess that a hair shader would be too useful. Nails are not in the shape of fibers, nor do they have the kind of complex occlusion and shadowing that hair does.

That being said, if the scattering properties of the nails are different enough from that of skin, it might make sense to use a different shader (for instance, doing some sort of volumetric scattering instead of a dipole approximation). I don't know of any references on skin scattering that have investigated nails specifically, though.


Ray tracing where illumination itself affects the geometry I am interested in a scientific application in which ray tracing is necessary, but also where the light itself alters the geometry of the surface mesh (imagine perhaps that the beams of light have enough energy to damage the surface). Right now I am writing everything from scratch and not using any existing libraries, and thus my (obviously naive) implementation is extremely slow. As I understand it, this boils down to clever methods by which to update the BVH, interleaved with each rendering step.
Some slight elaboration on the problem: when a ray of light hits a surface, we consider two possibilities: either it reflects off the surface (as is typical in a ray tracer), or it impacts the surface and essentially burns a hole into the surface. This alters the geometry  of the mesh and affects any future light rays, and thus forces an update of the boundary volume hierarchy. The surface deformation is usually local but can be global-scale, for example if the hole is large enough it may cause the structure to collapse or break off. Thus ray tracing affects the geometry "in real time", but simultaneously geometry is necessary to perform ray tracing at each step as well.
While I'm interested in the algorithmic problem, what I'm more interested in is whether or not there is a graphics library that is well suited to this task, so that I don't have to re-invent the wheel, or dive too deep into the literature. I am quite a beginner when it comes to available software libraries so please forgive me.

You might look into Mitsuba, which is a physically based path tracing renderer that's designed to be easy to extend with plugins. It might be useful as a base for you to build on, implementing your own algorithms as plugins or by modifying its code.
It already contains facilities for measuring the amount of light falling on some geometry, in the form of the "irradiance meter" object. I'm not sure how much it can help you with geometry modification as it's not really designed for that (it just imports geometry from a file and renders it), but maybe you can find some other libraries to help with that part, depending on what kind of geometry operations you need to do.


Morph Targets Normal adding math I am trying to verify that if I have my inNormal and some N amount of morph normals that I would want to do something such as

outNormal = vec3(inNormal);
outNormal += morphNormal0 * morphWeight0;
outNormal += morphNormal1 * morphWeight1;
...
outNormal += morphNormalN * morphWeightN;

This seems to work for me (don't have a ground truth to compare) so I wanted to test it in Three.js knowing it would have an established method and I see they use

objectNormal += ( morphNormal0 - normal ) * morphTargetInfluences[ 0 ];
objectNormal += ( morphNormal1 - normal ) * morphTargetInfluences[ 1 ];
objectNormal += ( morphNormal2 - normal ) * morphTargetInfluences[ 2 ];
objectNormal += ( morphNormal3 - normal ) * morphTargetInfluences[ 3 ];

So now I am wondering how that math works out as it doesn't work for me. Both methods end up multiplying it by a normalMatrixso not sure what would be different.

edit: Assuming both are loading in glTF 2.0 models

It would seem that, in the second example, the morphNormals are targets while in in the first they are deltas.

This explains the difference in math used between the two pieces of code.


Rendering in radiometric units or photometric? In traditional rendering, it is typical to do all the calculations with radiometric units, either as full spectral rendering, or component-wise (XYZ, RGB, etc).

However, as modern rendering adds more physically based models, it's convenient for artists to be able to specify values in the more familiar photometric units, for example, intensity of a light in lumens.

In order to keep the pipeline all in one type of unit, you can do one of the following:

Convert photometric units to radiometric units using luminous efficacy
Keep the entire render pipeline in photometric units

Sébastian Lagarde presents this idea very well in the course notes of his Siggraph 2014 presentation Moving Frostbite to PBR

My questions then are these:

Are there any disadvantages to rendering exclusively in photometric units?

As far as I can tell, photometric units are just radiometric units biased to the human eye. Considering that we will view the final image with our eye, I don't see this as a problem.

Frostbite is a RGB component-wise engine. Would a spectral renderer have any additional disadvantages by rendering exclusively in photometric units?

It's fine to use photometric units as an overall scale for setting light brightnesses. However, there's a technical subtlety you should be aware of. I'll quote from a blog post I wrote on the subject last year:

With RGB images, it’s important to recognize that our display devices behave more radiometrically than photometrically. A red pixel value of 255, and a green pixel value of 255, both result in about equal amounts of radiant flux (watts) being generated by a pixel on a screen—not equal amounts of luminous flux. By the same token, digital cameras capture pixel values that correspond to radiant flux, not luminous flux.
That’s why we need to use luma coefficients when converting images to grayscale, or calculating the brightness of a pixel, to get a perceptually accurate result; and it also means that rendering RGB images proceeds more naturally in radiometric units than in photometric ones.

In other words, the wavelength dependence of photometric units is different to what you might be expecting. In ordinary RGB color spaces, white is (1, 1, 1) and has a roughly flat radiometric spectrum; but in a supposed "photometric RGB", (1, 1, 1) would not be white; it would be a purpleish color, with less energy in the green range and more in the red and blue ranges. A similar problem would afflict spectral renderers trying to measure all their bins in wavelength-dependent photometric units, but even worse, as the radiance needed to generate a given luminance diverges toward either end of the visible spectrum, where the human luminous efficiency function goes to zero.
So, if you wish to use photometric units, IMO it's better to "cheat" a bit and not use true wavelength-dependent photometric units, but just use some fixed wavelength (such as 555 nm green, which is the peak of the human luminous efficiency function), or perhaps an average over the spectrum, as a reference unit, and apply that single unit to measure all wavelengths. This will get you in less trouble when importing RGB colors and spectra from other sources, and when generating them as outputs.


Gamma correction and halftone I'm writing a halftone algorithm, which takes standards RGB intensities in the range of $0-255$ and outputs black and white elements with a corresponding size ratio.

Should I apply a gamma expansion on the input luminance to get the ratio?

I.e. is it correct to do the following?

$\frac{w_\text{white}}{w_\text{white}+w_\text{black}} = (0.299 R + 0.587 G + 0.114 B)^{2.2}$

Should I apply the exponent of $2.2$?

Yes and no. The RGB values are most likely gamma compressed.

Consider gamma calibration images like the one below.

For a well calibrated monitor the gray value should look the same as the pattern of black and white lines.

If you load in the image you will likely find RGB values of $186$.

$(186/255)^{2.2} \approx 0.5$, which means that a halftone pattern of black and white lines of equal width correspond to RGB values of 186.

However, the gamma compression applied was more likely the sRGB gamma compression than an overall gamma of $2.2$.

$$
\frac{w_\text{white}}{w_\text{white}+w_\text{black}} =
\begin{cases}
12.92V  &\text{ if } V \leq 0.0031308 \\
1.055 V^{1/2.4} + 0.055 &\text{ if } V > 0.0031308
\end{cases}
$$

where $V = 0.299R+0.587G+0.114B$


What's the current state-of-the-art algorithm for ray-tracing height-fields? There have been lots of papers over the years on different techniques for drawing height-field terrain in a ray-tracer. Some algorithms ray-march the grid directly (or via a quadtree); others transform the terrain into a polygon mesh and use a standard ray-triangle intersection test. The research seems to have moved on in the last few years, and it's hard to find papers written in the last decade, but the balance between memory and compute (both CPU and GPU) is still changing.

What kind of algorithm gives best performance on high-end desktop machines nowadays? Or if there isn't a single answer, how do the performance characteristics of the current best algorithms differ?

For the current state-of-the-art, look for this paper:
"Maximum Mipmaps for Fast, Accurate, and Scalable Dynamic Height Field
Rendering", Tevs et al. 2008

The basic idea is to skip a lot of space by having knowledge of the maximum value over large areas of terrain. If the ray stays above that, skip to the next large area.

If you look at Figure 8, you'll see a comparison of basic linear stepping vs. maximum mipmaps. Linear stepping results in 200 steps, which can be done real-time on modern gpus but is still actually slow. Max mipmaps do the same in about 10 steps, all in shader.


Why should you do homogenization after clipping? Is it just so you avoid unnecessary divisions or are there other advantages as well?

I've just implemented the Sutherland–Hodgman clipping algorithm and read online that you should do homogenization after clipping to avoid divisions and to "avoid messy special cases". Now I wanted to ask if someone knows what they mean by "messy special cases".

One good reason to do clipping in homogeneous space is that the perspective division loses the distinction between regions behind the camera and in front of the camera. Points in front of the camera have $w > 0$, and behind the camera $w < 0$. After dividing by $w$, points behind the camera get mapped (with a sign flip) to the same $x, y, z$ NDC space as those in front of the camera. So it is no longer possible to tell the difference; the original sign of $w$ is lost.

This implies that at least the $w$ clipping must be done in homogeneous space. For instance, triangles that have one vertex behind the camera must get clipped by the $w = 0$ plane to form a quad. Otherwise, you end up with a bizarre object called an "external triangle"—a triangle that appears in two pieces, both extending to infinity, originating from the in-front-of-camera and behind-camera regions of the original homogeneous triangle.

And I suppose that once you have to apply one of the clipping planes in homogeneous space, you might as well apply all of them there.

For further reading on this, here are a couple of good articles:

In-depth: Software rasterizer and triangle clipping by Simon Yeung
A trip through the Graphics Pipeline 2011, part 5 by Fabian Giesen


How to get the GLFW_CONTEXT_VERSION_MAJOR value? This code snippet:

GLint versionMajor;
glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 2);
glGetIntegerv(GLFW_CONTEXT_VERSION_MAJOR, &versionMajor);
std::cout << "Version major:" << version << std::endl;

prints "3" in my screen, but in the first line I set the the GLFW_CONTEXT_VERSION_MAJOR to "2". How I can get back that value?

According to the GLFW docs, the context version hint acts as a minimum version, i.e. the context you actually get may be a higher version than what you request. However, the context should be backward-compatible, so even if you get a GL 3.x context you're OK to use GL 2.x code.


Physically Based Area Lights So after reading https://www.guerrilla-games.com/read/killzone-shadow-fall-creating-art-tools-for-a-new-generation (a guerilla games presentation on PBR) I noticed the Area lights as opposed to the standard Directional, Point & Spot lights in DirectX can produce very realistic shadows.

Now a lot of the resources online talk about it and what it is but there's not much regarding implementation. Some websites online say that it is just an array of lights but I don't think that's what it is.

Can somebody explain how I can implement this (and maybe with some code). If you could include any links to some good resources regarding implementation that would be great aswell.

When I implemented real-time area lighting, there were two documents I kept referring to: "Moving Frostbite to PBR" by Sebastien Lagarde and "Real Shading in Unreal Engine 4" by Brian Karis. Also "Lighting of Killzone Shadow Fall" by Michal Drobot is an interesting read on the topic. None of these documents deal with area shadows though but focus on energy conserving approximation of the lighting integral. While there are some code snippets, these documents lack quite a bit of details of how to implement area lighting though, so you need to do fair amount of research on your own. For example I posted a question and an answer here related to implementing rectangular area lights.
A very valuable tool I used was to implement a Monte Carlo reference model to easily compare different approximations to the ground truth. Some people also use Mitsuba renderer to validate their implementations. I think this ground truth reference is extremely important so that you know if there are mistakes in your implementation and also to see where/how these approximations fail and if you can somehow try to improve them. Implementing Monte Carlo integration is also quite a good exercise to understand the underlying problem.


Convolution of Hemispherical Harmonics Spherical Harmonics (SH) are a way to represent low-frequency spherical functions with only a handful of coefficients. They have some nice mathematical properties, e.g. a convolution with a kernel function h(x) (that has circular symmetry) can be calculated as

In the case of a convolution with a cosine lobe for rank 3 SH this results in a simple scaling of the bands with the factors

In many cases, e.g. incident light for a given point on an opaque surface, full spherical information is not needed, since half of the sphere is zero / undefined / unused anyway. Thus, Hemispherical Harmonics (HSH) were born.

How does convolution with an arbitrary kernel (with circular symmetry) work for HSH? Can the convolution from SH be extended or is there any paper that goes into details on this?

This answer tries to give a short overview over some important aspects. Since the HSH definition is rather complex and I couldn't find an overview over some pre-evaluated functions, I didn't provide examples simply because it would take me too much time right now.

Problem Description & Brute Force

To determine the convolution any with any set of basis function and thus computing the coefficients we generally need to compute the integral over the domain (= sphere for SH, hemisphere for HSH). Everything we need to do, to represent the hemispherical function f, which is defined over the angles theta ("up/down") and phi ("left/right"), via a coefficient c for HSH basis functions H is the following:

The sin(theta) is there because we integrate over the surface of a (hemi-)sphere. Conceptually, the size of a piece of area that comes from changing phi is bigger or smaller on the current theta. More on this here

If we don't care too much about accuracy or computing time we can solve this simply by sampling: Generate equally distributed (!) directions on the hemisphere, compute the product of f and H and average the outcomes (if you have truly equally distributed points you don't need the sin(theta)).

Get Started with an Analytical Solution

Of course we would love to have an analytical solution for our function, but this is where things can get very difficult. As a first step we may need to convert a function that is given on Cartesian directions into spherical coordinates. This part is still easy, just replace all your x, y and z as following:

Note that this gives us a system where the z-axis is the "up" of the hemisphere (theta=0) which should be represented by the HSH. After that it may already be possible to insert everything into a computer algebra system and solve the equation. Do not try to solve for all m & l but rather try one coefficient at a time, since it is unlikely that there is a compact expression that describes all of them at once. The definition of HSH is relatively complex, which makes it very tedious to evaluate these functions. In this paper the zero and 1st order HSH basis functions are mentioned in cartesian coordinates.

Notes on Rotations & Zonal Harmonics

Functions that are rotational symmetric around this z-axis are very good candidates for a successful analytic derivation, since they only affect the zonal coefficients, which are all coefficients with index m equals zero. This is especially helpful for the more general Spherical Harmonics where an easy formula exists that allows to rotate any Zonal Spherical Harmonics representation to an arbitrary direction, resulting in a Spherical Harmonics representation without any data loss (see here). This means that you can derive ZSH coefficients by assuming that your radial symmetric "function points to z" and rotate it then it into any desired direction. This works perfectly for example with various cosine lobe variations and also gives you the factors you mentioned in the question.

Now the bad news: For HSH, any rotation of a function around another axis than z is lossy, since your function will "touch" the lower undefined hemisphere after the rotation. Therefore, there is also no convenient "Hemi Zonal to HSH" rotation formula. Instead, there are multiple ways to do it with different drawbacks. For more details see the paper and the presentation.

By the way: All this is easier with the H-Basis, which is hemispherical as well (but originally only defined for a limited number of frequency-bands).


Is there different between 'if' and 'ternary operator' in GLSL? I'm practicing GLSL on ShaderToy
void mainImage( out vec4 fragColor, in vec2 fragCoord ) {

vec2 coord = fragCoord.xy/iResolution.xy;

vec3 background = vec3(1.);

vec3 axesColor = vec3(1.,0.,0.);
vec3 gridColor = vec3(0.85);

vec3 color = background;

const float tickWidth = 0.1;
for(float i = 0.0 ; i < 1.0 ; i += tickWidth)
{
    if(abs(coord.x - i) < 0.002) color = gridColor;
    if(abs(coord.y - i) < 0.002) color = gridColor;
}

if(abs(coord.x) < 0.005) color = axesColor;
**if(abs(coord.y) < 0.005) color = axesColor;**
**//color = abs(coord.y) < 0.005 ? axesColor : background;**

fragColor = vec4(color,1.0);  }

my question is on last 3 lines.
First, I used ternary operator for branch, but it made weird result that only drawing "x axes".
However, when I changed the operator to if, it works well. The code drew x, y axes and all grids as I wanted.
So I wonder is there different between if and ternary operation in GLSL?
If that, how can I choose correctly?

The difference is obvious. The ternary operator will return one of the two alternative values it is given; that's what it is for. An if condition that doesn't have an else block won't do anything if the condition is false.
The ternary operator version will set color to axesColor or background. But by that point in your code, you've gone past 3 other conditions that may have changed the value of color from its initial value of background. But the ternary operator doesn't care. It will overwrite whatever is in color with either axesColor or background.
What you probably wanted was color = abs(coord.y) < 0.005 ? axesColor : color;, so that the value will be preserved if the condition is false.


Given two 3d vectors, find the rotation matrix between the two So I am making a raytracer following the very helpful online book of Gabriel Gambetta but I stumble on the rotation matrix part.
My linear algebra background is 3Blue1Brown video series "The essence of Linear Algebra".
I simulate a camera placed at the origin that's looking forward ie whose direction is the vector {0,0,1}.
Let's say I am given another camera direction with direction {a,b,c}.
From what I understand if I find the rotation matrix R that takes as an input {0,0,1} and spits out a normalized {a,b,c} as an output.
Then I can apply that rotation matrix to all my rays in order to rotate them.
I can't get my head around on how to find R. Any help/explanation/correction much welcome.
I am in a similar situation than the person asking this question except that he seems to know already what needs be done and is just asking
about if his rationale is good.
Thanks
EDIT : The answer below is valid considering how I framed my problem but note that if you were like me trying to compute a camera to world rotation matrix and expected the viewport of your camera to follow along, you need an additional step because the answer below only gives you a shear matrix. In order to get a rotation matrix you need to compute the upVector with normalize(cross_product(rightVector, forwardVector)) (in a right hand system) and feed its coordinates into the matrix instead of {0,1,0}

You can construct a rotation matrix from an "axis", or 3 vectors. This is done by calculating 3 direction (normalized) vectors for the 3 axis of our new rotated coordinate system, they are forward, up and right vectors.
In your case let's say we have 2 vectors called v1 and v2.
we can produce a direction from them via (glsl psuedo code):
vec3 forwardVector = normalize(v2 - v1);

Then for our up vector we can pick the world-up axis (e.g. 0,1,0), although this may give you problems if the forward vector is very close to the up axis, in that case you can pick another arbitrary axis that's not nearby, like {0,0,1}. For the sake of simplicity we'll stick to the world up axis of {0,1,0}.
The last vector can be derived automatically from the other 2 vectors using a cross product.
so now we should have something like:
vec3 forwardVector = normalize(v2 - v1);
vec3 upVector = vec3(0,1,0);
vec3 rightVector = normalize(cross(forwardVector, upVector));

These 3 vectors can be directly plugged into a 3x3 matrix to form the rotation matrix.
r.x r.y r.z
u.x u.y u.z
f.x f.y f.z

where f=forwardVector, u=upVector and r=rightVector.
This is similar to an openGL "LookAt" matrix.
Be aware there are many rotation matrices that are valid solutions because 2 vectors cannot describe what the rotation is along the v1->v2 axis (That's what the 'up' axis fixes in this case).


Why are width and height divided by 2 in the perspective projection matrix? I'm using OpenGL to learn computer graphics.  I'm trying to understand projection matrices.  The following are shown in the OpenGL 'Red Book':

I don't understand why the width and height are divided by 2?

Because $x_{proj}$ doesn't vary from $0 \to width$, it varies from $-\tfrac{width}{2} \to \tfrac{width}{2}$.

What's important is not the width, but the minimum and maximum values of $x_{proj}$. Because $(0, 0)$ is in the centre of the viewport, not the corner, the min and max values are plus and minus half of the width.

BTW, the Red Book is not a great introduction to graphics in general. It's a technical standard, not a tutorial, so the explanation of concepts is just enough to be unambiguous - it's not trying to teach the material. It's also about a scanline rendering library in particular, so there's a lot of important concepts it doesn't describe at all, such as what BRDFs are important, how they relate to real-world surface properties, what an image really is, and many higher-level techniques.

And finally, even in the area it's about - how to use a GPU to accelerate rendering - it's hideously out of date. The GL architecture is based on GPUs from 20 years ago. Although lots of features have been added to keep up with changes to hardware, the fundamental abstractions are not a good way to understand current GPU architectures.

If you want to learn about the theory of computer graphics, start with a proper textbook such as Physically Based Rendering or Computer Graphics: Principles and Practice, or an online course. If you want to learn how to make a 3D application, start with a tutorial about the kind of application you're interested in (e.g. if it's a game, try a Unity tutorial).


How do I handle uniforms in a deferred renderer? I'm currently working on a renderer that tries to minimize the amount of state changes when drawing objects.
During each frame the client submits draw instances (the client provides a render state (shader program, vao, vbo) and the vertex offset and count).
At the end of each frame the renderer sorts these draw instances by depth and state and generates a list of draw commands (e.g. BindVertexBuffer, UseProgram, DrawArrays).
Now my question is, how would I go about handling uniforms in a renderer like this which holds on to all actual gpu communication until the end of the frame.

I have an idea on how to do this using callbacks which would be passed to the renderer when submitting a draw instance.
Something like this:

// Called several times by systems during the game loop
renderer.add(program, vbo, VertexPositionColor::getFormat(), 0, 3, [](){
    const GLfloat[4] color = { 1, 0, 0, 1 };
    glUniform4fv(0, color);
});
// Called once at the end of the game loop
renderer.draw(); // sort, generate commands, execute commands

But this seems like a less than optimal approach as it'll most likely trash the icache if there is significant amount of draw instances.
Would this be a valid approach? And what are some of the alternatives?

I'd say that the fundamental problem here is your approach. You basically want the outside renderer to throw a bunch of random stuff at the wall, then let the internal rendering system sort it out without having any idea what any of that stuff actually means.

Here's what I mean. Consider the case of skinned meshes vs. unskinned meshes. The internal rendering system has no idea which ones are skinned or not; all it knows is that some draw commands use a particular program and a large UBO/SSBO, while others use a different program and much smaller UBOs.

And that kinda works... to a point. And that point is when you're dealing with state values that are no longer encapsulated in objects.

Uniform values are program state. And you might reasonably have two different objects that use the same program have the same value for that uniform. They might have different values for other uniforms, but there would be different variance in which objects use which values.

But here's the thing: in pretty much every case where this happens, someone knows that it happened. And the code that knows that is the higher level code. After all, if two objects have common programs and common uniform values, they probably have a bunch of other stuff in common too. So odds are good that the higher level code knows when such things would happen.

By using this "throw stuff against the wall" approach, by separating the business end of your renderer from the source of that rendering data, you destroy the semantic information you could have used to detect such things. That is, it is the higher level system that should handle the "sort by state change" part, since it can do a much better job.

However, given that you've effectively designed yourself into a corner, I'd say that the simplest approach is to develop a way to encode uniform values, rather than relying on callbacks.

Uniforms can be scalars, vectors, or matrices (arrays and structs use multiple uniform locations). So you need the equivalent of a "variant" type that can store all of such data, of the types supported by GLSL. And then, for each object, you walk through each uniform location in that object's data block and set that value. Sure, "visitation" is not the most instruction-cache friendly approach, but it's a lot better performing than calling a std::function. At least with visitation, you're executing the same block of code each time.

So what you'd have is a (possibly dynamic) array of uniform data, where each uniform datum is a location, a type, and the variant data. I'll leave the storage optimization of the "variant" as an exercise for the reader.

Generally speaking however, non-block uniform usage should be fairly minimal, dealing only with values that change frequently while rendering with a program. So even in that case, things shouldn't be too bad.


Finding the normals of a tileable 2D surface extracted from 4D space If I use a 4D noise function which returns a noise value and a four-part analytically derived gradient vector, I can create a 2D tile which is seamless (i.e. the opposite edges match) through remapping x,y and z,w coordinates to independent circles, i.e.

$\ \theta = a*2*\pi$
$\ \phi = b*2*\pi$

where $a$ and $b$ are a point on the surface. They should be within the range $[0, size)$ where size is some arbitrary shared dimension for the surface.

We can then plug $\theta$ and $\phi$ into the below functions to get the equivalent point in 4D space.

$\ x = r_0*\cos(\theta)$
$\ y = r_0*\sin(\theta)$
$\ z = r_1*\cos(\phi) $
$\ w = r_1*\sin(\phi)$

Effectively the surface created through this coordinate mapping is a flat torus.

However, since the noise function operates in 4D space I'm left with a 4D gradient vector. Is it possible to transform this 4D vector into a 2D vector which describes the gradient of the surface?

Since the question was somewhat clarified I will formalize both the question and the answer for future readers.
Having a differentiable scalar field $f : \mathbb{R}^4 \rightarrow \mathbb{R}$ we want to find the gradient of the field with respect to $\theta, \phi$ on the 2-manifold defined parametrically by:
$$(x(\theta,\phi), y(\theta,\phi) z(\theta,\phi), w(\theta,\phi)) = (r_0\cos\theta, r_0\sin\theta, r_1\cos\phi,r_1\sin\phi)$$
Where $r_0,r_1$ are constants. We can compute the partial derivatives as follows:
$$\frac{\partial f}{\partial \theta} = \frac{\partial f}{\partial x}\frac{\partial x}{\partial \theta} + \frac{\partial f}{\partial y}\frac{\partial y}{\partial \theta} + \frac{\partial f}{\partial z}\frac{\partial z}{\partial \theta} + \frac{\partial f}{\partial w}\frac{\partial w}{\partial \theta} \\
= -r_0\frac{\partial f}{\partial x}\sin\theta + r_0\frac{\partial f}{\partial y}\cos\theta$$
$$\frac{\partial f}{\partial \phi} =
\frac{\partial f}{\partial x}\frac{\partial x}{\partial \phi} + \frac{\partial f}{\partial y}\frac{\partial y}{\partial \phi} + \frac{\partial f}{\partial z}\frac{\partial z}{\partial \phi} + \frac{\partial f}{\partial w}\frac{\partial w}{\partial \phi} \\ = -r_1\frac{\partial f}{\partial x}\sin\phi + r_1\frac{\partial f}{\partial y}\cos\phi$$

Assuming that the gradient $\nabla f = (\frac{\partial f}{\partial x},\frac{\partial f}{\partial y},\frac{\partial f}{\partial z},\frac{\partial f}{\partial w})$ is known, it is straightforward to compute the 2d gradient.

Note that another possibility is to compute the gradient numerically (in case you don't have the partial derivatives of $f$):
$$ \frac{\partial f}{\partial \theta} \approx \frac{f(r_0\cos(\theta+h_{\theta}), r_0\sin(\theta+h_{\theta}), r_1\cos\phi,r_1\sin\phi) - f(r_0\cos\theta, r_0\sin\theta, r_1\cos\phi,r_1\sin\phi)}{h_{\theta}}$$
$$ \frac{\partial f}{\partial \phi} \approx \frac{f(r_0\cos\theta, r_0\sin\theta, r_1\cos(\phi+h_{\phi}),r_1\sin(\phi+h_{\phi})) - f(r_0\cos\theta, r_0\sin\theta, r_1\cos\phi,r_1\sin\phi)}{h_{\phi}}$$
For $h_{\theta}, h_{\phi}$ chosen adequately (note that the choice of these parameters is a nontrivial problem).


About the use of Russian Roulette in Smallpt I was taking a look at Smallpt (http://www.kevinbeason.com/smallpt), more specifically at the Russian Roulette part. Actually, RR is used in two places along the code: first, to determine ray termination; and second, for the rendering of dielectrics.

That is the code for the ray termination case (expanded code version):

// p = maximum reflectance
// fr = BRDF
double p = fr.x > fr.y && fr.x > fr.z ? fr.x : fr.y > fr.z ? fr.y : fr.z;

if ( ++depth > 5 )
    if ( erand48(Xi) < p )
        fr = fr * ( 1 / p );
    else
        return obj.emission;

After five bounces, RR is used to determine if the ray path continues or not. If it continues, the BRDF is scaled to compensate for it. This is Ok for me.

RR is also used in a second code segment to select between refraction or transmission during the rendering of dielectrics (expanded code version):

double Re = R0 + ( 1 - R0 ) * cost * cost * cost * cost * cost;
double Tr = 1 - Re;
double P = .25 + .5 * Re;
double RP = Re / P;
double TP = Tr / ( 1 - P );
Vec Lo; // output radiance

if ( depth > 2 )
    if ( erand48(Xi) < P )
        Lo = radiance(reflRay,depth,Xi) * RP;
    else
        Lo = radiance(Ray(x,tdir),depth,Xi) * TP;
else
    Lo = radiance(reflRay,depth,Xi) * Re + radiance(Ray(x,tdir),depth,Xi) * Tr;

According to the code above, the path branches recursively if the ray has bounced up to two times. After two bounces, however, RR is used to select the path to be followed. This is also Ok for me.

What is a bit confusing is the fact that the radiance returned by both possible non-branching paths (refraction and transmission) is scaled. I understand that there are different probabilities regarding reflection and transmission. However, if for instance Re = 0.3 and Tr = 0.7, and 100 rays strike the surface, about 30% of the rays will be reflected and 70% of will be transmitted due RR. In this case, I understand that there is no path termination neither energy loss, so there wouldn't be anything to compensate for.

Thus, my first two questions are: why are these radiances scaled? Should they be scaled, or would it work without scaling at all?

My third question is related to the scaling factors: Why the author has used P, RP and TP instead of Re and Tr?

Any indication of a good reading about this topic is also very welcome!!

Thank you!

The reflected and transmitted values need to be scaled because they are not sampled proportionally to their contribution. You'll notice how the decision whether to reflect or transmit is made based on P, not Re.

About why that scaling has been done in the first place, only the author can give you the definitive answer. My guess is that this is done to prevent extreme values that would come from tiny probabilities. Imagine that Re is 0.001 and is used without scaling. Then only one in 1000 rays would reflect, and the contribution of that ray would be multiplied by 1000. At a render with, say, 64 rays per pixel, this would leave very visible artefacts, as not every pixel would get a reflected ray. The scaling from this code snipped on the other hand makes sure that the ray budget is more evenly distributed between reflection and transmission and that each potential path taken is guaranteed at least 25% or the rays.

For further reading about path tracing, there is no better and comprehensive resource than www.pbrt.org - both the book and the source code are worth reading.


Shader sampler2D uniforms instead of attributes In the godot game engine (using opengl) it is not possible to define custom shader attributes.
A workaround is creating a sampler2D texture uniform instead and use vertexId (or in fact UV since vertexID also isn't exposed) to get the per vertex attributes.

I am wondering are there any downsides to this approach and/or caveats one should keep in mind ?

I copied my answer for the same OP at https://github.com/godotengine/godot/issues/19473. But this question differs slightly to the original issue report.
In case you want to manually add support for VERTEX_ID, these are the changes I had to make in 3.2.3-stable for spatial shaders. I wanted to use the vertex ID for creating procedural animations.
In drivers/gles3/shader_compiler_gles3.cpp in the function ShaderCompilerGLES3::ShaderCompilerGLES3() :
actions[VS::SHADER_SPATIAL].renames["VERTEX_ID"] = "gl_VertexID";

and servers/visual/shader_types.cpp in the function ShaderTypes::ShaderTypes():
shader_modes[VS::SHADER_SPATIAL].functions["vertex"].built_ins["VERTEX_ID"] = constt(ShaderLanguage::TYPE_INT);

Then compile the source and it should work. Some other primitives could probably be added in a similar fashion but I have not tested that yet.
One caveat is that the vertex IDs reported by Godot are very different from those reported by other tools such as Blender. This is because vertices are duplicated and in a different order. Godot also only gives you two UV channels to work with. Using UV coordinates also means having a temporal component will require significantly more memory to implement as UVs will typically map sparsely onto a 2D texture, if the data is only used in the vertex shader.
I have a primitive tool working that can map the vertex animation texture generated in blender to one usable by Godot into another grayscale texture (n by 1 texture, where n is number of vertices in Godot). This saves texture memory as no data is duplicated for duplicated vertices. I first need to implement an octree before it will be usable for real-world meshes and not just a set of simple deforming cubes. Once I have that done in my spare time I will release it for free.


Implicit conversion in glsl shaders When specifying member values for a structure consisting of float values like vec3 or mat3, is it guaranteed that if an integer value is passed it's going to be promoted to a float? For example if I write

vec3 var = vec3(1, 0, 0)

Same when assigning an integer value to a float variable:

float var = 1

Is the implicit conversion guaranteed here?

Yes, it is guaranteed by the GLSL language specifications (click me). Section "4.1.10. Implicit Conversions" says

  In some situations, an expression and its type will be implicitly converted to a different type.

The spec shows all possible implicit conversions inside a table, also int -> float. It further says:

  When an implicit conversion is done, it is the same conversion that would be done under explicit
  conversion, using a constructor.

e.g.

float a = 1;

is equivalent to writing

float a = float(1);

There is one sentence at the end of the section, that you should keep in mind:

  The conversions in the table above are done only as indicated by other sections of this specification.

If read it correctly, it says that you can not rely on implicit conversions in any kind of instruction you might come up with. However, regarding vec3 var = vec3(1, 0, 0) section "5.4.2. Vector and Matrix Constructors" says:

  If the basic type (bool, int, float, or double) of a parameter to a constructor does not match the
  basic type of the object being constructed, the scalar construction rules (above) are used to convert
  the parameters

So it calls the constructor float(int) before consuming each value.

Even though both cases are explicitly backed by the language specifications, my personal advice is to never rely on implicit conversions. They can lead to subtle, hard to track bugs. Not sure how it is in GLSL, since I didn't test it, but in C++ this one:

float a = 1 / 2; # <--- this evaluates to 0.f!

, deeply hidden somewhere in larger calculations, has cost me several hours of debugging more than once. It gets worse if one or both values of the fraction are taken from a variable. However, in C++ I can step through the code with a nice debugger, something that is not so easily done in GLSL. So if the only reason for using implicit conversions is laziness or some sort of taste-related decision, force yourself to be explicit. It will safe you a lot of pain.


Program to make "Pieslice" without using library function? As we know in C/C++ there is "graphics.h" header file with the help of which we can do graphics programs in C.
There is a function pieslice in that header file,its syntax is:

#include <graphics.h>

void pieslice(int x, int y, int stangle, int endangle, int radius);

[x,y are the center of the circle,stangle and endangle are the starting and end angles respectively]

Can we make a pieslice in C/C++ without using this in-built function.Please help. Tried making it with the help of the lines and arc function of the
graphics.h but i am still not able to figure out how to connect the lines with the arc.
Thank You in advance.

My code so far:

#include<stdio.h>
#include<graphics.h>
#include<math.h>

int main()
{
    int gd=DETECT,gm;
    initgraph(&gd,&gm,NULL);
    int xc,yc,r,st_angle,ed_angle,StartX,StartY;
    printf("Enter the centers of pieslice:\n");
    scanf("%d %d",&xc,&yc);
    printf("Enter the radius:\n");
    scanf("%d",&r);
    printf("Enter the starting angle:\n");
    scanf("%d",&st_angle);
    printf("Enter the end angle:\n");
    scanf("%d",&ed_angle);

    double startRadians = (double)st_angle * M_PI /180.0;
    double endRadians = (double)ed_angle * M_PI /180.0;

    for(int k=0; k<startRadians;k++)
    {
        startX = xc+ cos(startRadians) * r;
        startY = yc+ sin(startRadians) * r;
    }
    for(int l=o; l<endRadians; l++)
    {
        endX = xc+ cos(endRadians) * r;
        endY = yc+ sin(endRadians) * r;
    }

You can use the sine and cosine functions to do this. Normally they start on the x axis and rotate counter-clockwise. The start point can be found by calculating:

startX = x + cos(startRadians) * radius;
startY = y + sin(startRadians) * radius;

Likewise the ending coordinates are:

endX = x + cos(endRadians) * radius;
endY = y + sin(endRadians) * radius;

Note that sin() and cos() take their parameters in radians. Given that stangle and endangle are integers, I'm guessing they're probably in degrees so you'll need to convert them by doing something like:

double startRadians = (double)stangle * M_PI / 180.0;

etc.

If you want to draw the entire pie wedge, you'll need to iterate from the start angle to the end angle and calculate every n degrees along the curve.


Get data from javascript There is an web-interface that load geometric models and show it inside the bounding volume. WebGL is used to generate it. So far no scene graph technique is used to organize the elements in the scene. Is there any javascript api that does support the scene graph technique ?

If there is, I am also required to know more if there is any mechanism to transfer the scene graph related information from the javascript to C++ based API. As of right now the scene that is developed without scenegraph technique stores that scene as an .stl file and C++ based API read the .stl file and process further. I believe that it is not the optimal way to deal with it. It would have much faster and optimal if we have the scene graph system support in javascript to organize the scene and the direct scene related data transfer from javascript to C++ for further processing.

Let me know if I have not explained the issue well enough.

Thanks

There does not seem to be a single answer for this question, but I will try to give some directions in possible solutions. First of all I presume that "the scene graph technique" means storing the object data in some kind of data structure (I know what a scenegraph is, but there are many different possibilities for indoor/outdoor scenes etc). There seem to be some Javascript libraries, but they do much more then simply providing the datastructure.

For defining the scenegraph structure you could look at glTF(https://github.com/KhronosGroup/glTF) defined by Khronos themselves. This really depends on what kind of features your scene graph requires.

If the C++ library is required for the processing you can look into providing Javascript bindings for it. You can either try to compile the library to javascript using emscripten (https://github.com/kripken/emscripten). Otherwise provide a C-interface and call it directly using ccall(). This way you can directly share this datastructure via parameters.


Spectral path tracing - image color/brightness incorrect I implemented a spectral path tracing using physically base BRDF models such as Oren-Nayar,Specular Reflection and Transmission, Lambertian. All calculation in the path tracer uses standard illuminant and macbeth color checker SPD, spectral power distribution. The result of the path tracer for each pixel is the SPD obtained as a sum of SPD obtained from each sample calculated by the pat tracer. This SPD is then converted to CIE XYZ color and to RGB. The result scene obtained is the following one (in this example taking 500 samples per pixel):

As you can see, everything seems fine, except for the brightness/luminance of the scene. Every object in the scene is darker than it have to be. The floor and the front wall of the cornell box in the scene must be white and neutral8 (from macbeth color checker), but they are dark gray.
The following method is the one that trace the samples for a pixel of the path tracer:

Vector3D PathTracer::getPixelColor(const Ray& ray, int bounce) {

    Spectrum<constant::spectrumSamples> L(0.0f);
    int numberOfSamples = 500;
    float sampleWeight = 1.0f/(float)numberOfSamples;

    for (int i = 0; i < numberOfSamples; i++) {

        Spectrum<constant::spectrumSamples> spectrumSample = trace(ray, bounce);
        L = L + spectrumSample * sampleWeight;
    }

    Spectrum<constant::spectrumSamples> Li = scene->light->spectrum;

    ColorMatchingFunction* colorMatchingFunction = new Standard2ObserverColorMatchingFunction();

    //Get tristimulus values.
    Vector3D tristimulus = CIE1931XYZ::tristimulusValues(L, Li, colorMatchingFunction);

    //Convert tristimulus to sRGB.
    Vector3D color = CIE1931XYZ::tristimulusTosRGB(tristimulus);

    //Apply sRGB gamma correction.
    sRGB::sRGBGammaCorrection(color, GammaCompanding);

    //Convert to standard 0 - 255 RGB value.
    sRGB::sRGBStandardRange(color);

    delete colorMatchingFunction;

    return color;
}

As you can seen, I already apply a gamma correction to the color obtained.
Do you have any idea why my image rendered is so dark? My concerns are in the part where I convert the SPD sum, obtained from the samples, into RGB color. Do you see any error? Am I missing something? Do I need other operation to execute a correct conversion from the SPD obtained from the sampling to an RGB color?

To avoid to write a too long question, I will link the main classes used by the path tracer for the calculation:

Lighting/shading class used to recursive calculate the rendering equation https://github.com/chicio/Spectrum-Clara-Lux-Tracer/blob/luminance/SCLT/Shading/PathBRDF.cpp
CIE XYZ conversion https://github.com/chicio/Spectrum-Clara-Lux-Tracer/blob/luminance/SCLT/Color/CIEColorSpaces/CIEXYZ.cpp
sRGB utils https://github.com/chicio/Spectrum-Clara-Lux-Tracer/blob/luminance/SCLT/Color/sRGB.hpp

The other files/classes used are all on this repository (branch luminance)

https://github.com/chicio/Spectrum-Clara-Lux-Tracer/tree/luminance

Thanks you all guys, I hope someone could help me.

The problem lies mainly in CIE1931XYZ::tristimulusValues() function, where you normalize the resulting color to the luminance of your illuminant which causes that directly observed light source has luminance 1, but everything else is much darker. That is a nice thing to do if you just want to visualize colours of various reflectance spectra under a given illumination, but is probably not the best thing to do in a global illumination renderer.

To be more specific, you cannot expect a reflectance profile to be rendered always completely white just because it reflects all incident energy (constant spectral reflectance equal to 1 everywhere). Neglecting the material directional properties (BSDF, etc.), the properties of the light reflected from a point depend both on reflectance spectral profile of the material, and on the spectral and angular properties of the incident light. The incident light therefore depends not only on the SPD of the illuminant used for your light source, but also on its size and distance from the shaded point. The points closer to the light source will be brighter, while distant points will be darker.

I would suggest removing the luminance normalization in your Spectrum-to-XYZ conversion completely and then adjusting the overall brightness of your scene by multiplying the normalized SPD of the used illuminant (D65 in your case) with some brightness parameter. Whether you multiply it before rendering, during rendering or during the XYZ conversion step is completely up to you. The choice shouldn't affect the result.


2D projection from some points I am going through an undocumented function that takes in three points and a Z-value that projects three points on the plane defined by Z-value. I want to understand the mathematical theory behind it and I need your help to decode the function:

SlicerSegment project2D(Point3& p0, Point3& p1, Point3& p2, int32_t z) const
{
    SlicerSegment seg;
    seg.start.X = p0.x + int64_t(p1.x - p0.x) * int64_t(z - p0.z) / int64_t(p1.z - p0.z);
    seg.start.Y = p0.y + int64_t(p1.y - p0.y) * int64_t(z - p0.z) / int64_t(p1.z - p0.z);

seg.end.X = p0.x + int64_t(p2.x - p0.x) * int64_t(z - p0.z) / int64_t(p2.z - p0.z);
    seg.end.Y = p0.y + int64_t(p2.y - p0.y) * int64_t(z - p0.z) / int64_t(p2.z - p0.z);
    return seg;
}

The function calculates the intersection points between two edges of a triangle and a plane that is assumed to cut it. It returns a line segment running along the cut.

Or more generally, the function calculates where the intersection points with the two edges would be, if the edges were extended to infinite lines. Given that it's called "project", that might be the intended use-case; hard to know without seeing how this gets used in context.

The first two lines of math are calculating the intersection point of the $p_0 p_1$ edge with the z-plane. It works by linearly interpolating the $x, y$ values along the edge based on the proportion between $z$ values. The second two lines do the same thing for the $p_0 p_2$ edge.

One odd thing is that the function is using integers everywhere. Presumably it's because the program is using fixed-point coordinates (which makes good sense for a lot of geometry processing applications due to its uniform precision). However, the way it's done here seems oblivious to rounding: the integer divisions will truncate any fractional component instead of rounding to nearest.


Why do identical polygons not z-fight in OpenGL? Typically when coplanar or nearly coplanar polygons are rendered in OpenGL, the textures of each polygon mix together in a non-deterministic way. My understanding is that this is caused by the limited precision of the z-buffer as it uses floating point values. However, there seems to be an exception to this case. I've noticed that when two polygons are rendered with the exact same vertex positions, z-fighting does not occur. The polygon that was rendered last appears on top of the other coplanar polygon.

Why does this occur? Does this behavior always occur? Is this behavior guaranteed across all OpenGL implementations?

There are explicit rules in the OpenGL specification about invariance guarantees. For the purpose of this discussion, these rules boil down to the following.

Given a particular series of vertex processing shaders (VS, tessellation, and GS), there are some set of input values, uniforms, and the like which lead to the computation of the final gl_Position value for the primitives in that process. If all of these values are binary identical (including the way the vertex format leads to any VS inputs), then you're guaranteed that gl_Position will have binary identical values. And if two primitive of the same type have binary identical positions, the rasterizer is required to generate binary identical fragment positions.

And thus, no z-fighting.

Now, that's kind of restrictive, as it only works with the same shaders. You can use different shaders, so long as:

all shaders involved get binary identical values on expressions leading to the computation of gl_Position,
all shaders involved use the exact same expressions (and I mean exact; a + b isn't the "same expression" as b + a, no matter what mathematics says) to compute gl_Position, and
gl_Position and any inter-shader in/out variables leading to its computation are qualified with the invariant qualifier in all shaders. This tells the compiler not to optimize certain things away in expressions leading to the computation of such values.