Ambient occlusion of a sole object in a scene So, the german wikipedia states about ambient occlusion:The ambient occlusion method is based on the assumption that an object, a set of objects or an entire scene is surrounded by a unit sphere of light.

Now, let's suppose we've got a scene in which the only object is a pipe, closed at one end, at the center of the scene. If I got this right, then to get the ambient occlusion this object would now be evenly light from every direction.

But wouldn't lighting the object evenly from everywhere mean that for every point in the pipe there would also be a ray of light that would directly reach it, leading to no ambient occlusion at all (no shadows inside the pipe)?

What did I get wrong here?

When you compute ambient occlusion (AO) for a point on an object's surface, you sample multiple directions and measure what fraction of them escape. That is, you're trying to approximate what fraction of the unit hemisphere is visible to that point on the surface. (Note that it's a unit hemisphere because half of the unit sphere is on the wrong side of the surface to start with.)

So in your pipe example, the outside of the pipe is not at all occluded. From every point on the outside, any ray fired in that hemisphere will escape without intersecting the object. All of the outside would be lit with 100% of the ambient light.

The inside is another matter. The inside of the end cap can only see a small circle of that hemisphere, through the other end of the pipe. The exact fraction depends on the radius and length of the pipe, but call it 10% for the example. 90% of rays fired from a point on the end cap will hit the inside of the cap, and the other 10% will escape along the length of the pipe and out of the other end. These points are 90% occluded, so they'll be lit with 10% of the ambient light.

Consider a point at the open end of the pipe, but on the inside. This point is 50% occluded. Half of the rays fired from it go into the pipe and intersect with it at some point. The other half go straight out and escape to the unit hemisphere. This point will be lit with 50% of the ambient light, and it's easy to see that the 50% will smoothly decrease to the 10% (or however much is the minimum) as you travel down the inside of the pipe towards the end cap, as less of that hole is visible.

The important point I think you misunderstood is that you measure or estimate what fraction of the unit hemisphere is visible from the point you're shading, not whether it's visible at all. Casting many rays and counting what fraction escape is one way of estimating this.


Plotting pbrt-v3 Gaussian filter I want to plot the Gaussian filter of pbrt-v4 with a radius $(100,100)$ and a standard deviation ($\sigma$) of $0.5$.
I would expect something like this:

When I plot the pbrt-v4 Gaussian filter with radius $(100,100)$ and $\sigma=0.5$ I get:

Is the 2nd plot correct? I varied the radius form $1$ to $1000$ but the the filter size changes only 2 pixels in total (I get a similar results).
Any ideas?
How does the correct plot look like?

You probably want to set sigma to be a fraction of the radius, rather than a fixed value. Sigma controls the actual shape of the filter, while radius just controls how far out the filter gets cut off (as Gaussians truly have infinite radius, so we have to cut them off somewhere to use them in practice). With sigma set to 0.5, you're making a filter that's very sharply peaked within a radius of 0.5 pixels, which is consistent with the image you're getting.
Instead, try setting sigma = 0.25 or 0.3 times the radius. Or alternatively, make sigma your main parameter and then set the radius to 3 or 4 times sigma.


OpenGL back and front buffer - counting and adjusting the number of red pixels before displaying to screen I'm writing a function that reads the number of red pixels on the left and right halves of the screen, it makes adjustments while the difference between the number of red pixels falls outside of a threshold. I'm having trouble with the buffers (back and front) because when I finally swap to the front buffer, it dynamically draws all changes I made in the loop meant for the back buffer but I only want the final one drawn to screen. In pseudocode,

do {
    draw scene to back buffer
    read number of left, right red pixels
    if (abs(left-right)>threshold) {
        //some function to jitter objects in the scene
    }
    else done=true,
}while(!done);

//draw to front buffer

Here is some C++/OpenGL code:

do {
    glDrawBuffer(GL_BACK_LEFT);
    display(0);

    glDrawBuffer(GL_BACK_RIGHT);
    display(1);

    redsleft = 0;
    redsright = 0;

    glReadBuffer(GL_BACK_LEFT);
    glReadPixels(0, 0, resolution.first, resolution.second, GL_RGBA, GL_UNSIGNED_BYTE, frameBuffer);
    for (int j = 0; j < resolution.first*resolution.second * 4; j += 4)
    {
        if (frameBuffer[j]>0 && frameBuffer[j + 1] == 0 && frameBuffer[j + 2] == 0 && frameBuffer[j + 3] > 0)
        {
            if (j % 4 == 0 && i == 0)
            {
                int redX = (j / 4) % resolution.first;
                int redY = j / 4 / resolution.first;

                if (redX < resolution.first / 2)
                    redsleft++;
                else
                    redsright++;
            }
        }
    }
    //if above threshold, call function to adjust
    //else done
} while (!done);

glDrawBuffer(GL_BACK_LEFT);
display(0);

glDrawBuffer(GL_BACK_RIGHT);
display(1);

glutSwapBuffers();
glutPostRedisplay();

Any ideas?

I have tried doing glCear(GL_BACK_LEFT) between each iteration but it is not clearing, I'm wondering if I'm missing a line of code before the glClear, or if I'm calling it right. I think the problem is that the back buffer isn't clearing

This comment of yours sheds some light on the matter. This function call doesn't do what you think it does and is actually wrong usage. The glClear function doesn't actually take the buffer identificator (that you give to glDraw/ReadBuffer) to clear rather than a mask of the general parts of the framebuffer to clear.

The framebuffer usually consists of one or more color buffers, a possible depth buffer and a possible stencil buffer. Therefore you give glClear an ORed combination of the respective flags GL_COLOR_BUFFER_BIT, GL_DEPTH_BUFFER_BIT and GL_STENCIL_BUFFER_BIT and those buffers (of the currently specified draw buffer for the color buffer) are then cleared.

So after setting the draw buffer with glDrawBuffer you call glClear to clear that. As said, the flags depend on what you want cleared (usually at least GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT, special algorithms notwithstanding).


Why normalise Lambertian BRDF by 1/pi? Why is a Lambertian BRDF normalised by dividing by $\pi$? Since the area of a unit sphere is $4 \pi$, and the area of the half sphere above the surface is $2 \pi$, shouldn't it rather be $1/(2\pi)$?

I think I got it!

Because $cos(\theta)$ integrates to $\pi$ over the hemisphere (and not $2\pi$). And the incoming light is multiplied by $cos(\theta)$ (and the BRDF).


Mix shader looks wrong on my path tracer I apologize if my methods seem way off because this is my first time trying to build a path tracer and I'm struggling quite a bit. Currently, I am trying to mimic the "mix shader" node in Blender 3D that combines two different shaders. I would like to add this functionality to my path tracer because it would allow me to make a plastic-like material. Reading through a few explanations, it seems it takes a probabilistic approach to picking one or the other when a ray hits the surface.

Here is a rendering of an object with a 50/50 mix of ideal reflection and ideal diffuse at 80 samples per pixel:

Something is just off about it. Especially those weird underside highlights on the leg extending farthest to the left. I have a strong feeling I'm implementing it wrong.

Here is a pseudo implementation of the path tracing part. It uses explicit light sampling. Based off of smallpt path tracer.

rayColor(ray r, depth, int E=1)
{
    if(r doesn't hit triangle)
        return 0
    if(r hit is a light)
    {
       if(E)
         return light_emission
       else
         return 0
    }

    vector x = r.origin + r.direction*t // x is point where r hit tri
    vector n = normal where ray hit triangle
    n.normalize()
    vector nl = n.dot(r.d) < 0 ? n : n*(-1) // properly orient normal

    if(++depth > 5) return 0 // max bounces

    float triangle_area   = area of emitting triangle
    vector x_light_random = random point on emitting triangle
    vector light_normal   = normal of emitting triangle

    vector d = x_light_random = x_converted;

    if(light_normal.dot(d) > 0) light_normal *= -1; // make it emit
                                                    // both sides

    object_normal.normalize();
    light_normal.normalize();

    BRDF = 1/PI // perfect diffuse
    light_emitted = 1 // emission of 1
    vector light_out = 0
    if(ray starting at x towards d hits light (i.e. not in shadow))
    {
         light_out = BRDF*light_emitted*(object_normal.dot(d))*
                            (-1*light_normal.dot(d)*triangle_area)/
                            (d.length*d.length*d.length*d.length)
    }
    vector direct_light = color_of_object_triangle*light_out;

  //----SPECULAR-----
  vector d2 = r.dir-n*2*n.dot(r.dir); // ideal reflection
  vector light_color = 1 // white since dialectics don't change spec
  vector specular = light_color*rayColor(createRay(x, d2),depth)

  float P = 0.5; // 50/50 chance of mirror/diffuse
  if(erand(Xi) < P)
      return direct_light/P
  else
      return spec/(1-P)

Thank you to anyone who can offer me some direction! Again, sorry if this is really convoluted. I've made it so far and I'm about to give up, but I feel like I'm reaaaally close.

As suggested in the comments, I created an exact scene in Blender using the same .obj. After adding a mix shader node and setting one side to diffuse and the other to a perfect mirror, you can see that the results are almost identical. This would indicate that my implementation is correct.

Here is a side by side comparison. Mine is on the left, Blender's is on the right:


Should new graphics programmers be learning Vulkan instead of OpenGL? From the wiki: "the Vulkan API was initially referred to as the 'next generation OpenGL initiative' by Khrono", and that it is "a grounds-up redesign effort to unify OpenGL and OpenGL ES into one common API that will not be backwards compatible with existing OpenGL versions".

So should those now getting into graphics programming be better served to learn Vulkan instead of OpenGL? It seem they will serve the same purpose.

If you're getting started now, and you want to do GPU work (as opposed to always using a game engine such as Unity), you should definitely start by learning Vulkan. Maybe you should learn GL later too, but there are a couple of reasons to think Vulkan-first.

GL and GLES were designed many years ago, when GPUs worked quite differently. (The most obvious difference being immediate-mode draw calls vs tiling and command queues.) GL encourages you to think in an immediate-mode style, and has a lot of legacy cruft. Vulkan offers programming models that are much closer to how contemporary GPUs work, so if you learn Vulkan, you'll have a better understanding of how the technology really works, and of what is efficient and what is inefficient. I see lots of people who've started with GL or GLES and immediately get into bad habits like issuing separate draw calls per-object instead of using VBOs, or even worse, using display lists. It's hard for GL programmers to find out what is no longer encouraged.
It's much easier to move from Vulkan to GL or GLES than vice-versa. Vulkan makes explicit a lot of things that were hidden or unpredictable in GL, such as concurrency control, sharing, and rendering state. It pushes a lot of complexity up from the driver to the application: but by doing so, it gives control to the application, and makes it simpler to get predictable performance and compatibility between different GPU vendors. If you have some code that works in Vulkan, it's quite easy to port that to GL or GLES instead, and you end up with something that uses good GL/GLES habits. If you have code that works in GL or GLES, you almost have to start again to make it work efficiently in Vulkan: especially if it was written in a legacy style (see point 1).

I was concerned at first that Vulkan is much harder to program against, and that while it would be OK for the experienced developers at larger companies, it would be a huge barrier to indies and hobbyists. I posed this question to some members of the Working Group, and they said they have some data points from people they've spoken to who've already moved to Vulkan. These people range from developers at Epic working on UE4 integration to hobbyist game developers. Their experience was that getting started (i.e. getting to having one triangle on the screen) involved learning more concepts and having longer boilerplate code, but it wasn't too complex, even for the indies. But after getting to that stage, they found it much easier to build up to a real, shippable application, because (a) the behaviour is a lot more predictable between different vendors' implementations, and (b) getting to something that performed well with all the effects turned on didn't involve as much trial-and-error. With these experiences from real developers, they convinced me that programming against Vulkan is viable even for a beginner in graphics, and that the overall complexity is less once you get past the tutorial and starting building demos or applications you can give to other people.

As others have said: GL is available on many platforms, WebGL is a nice delivery mechanism, there's a lot of existing software that uses GL, and there are many employers hiring for that skill. It's going to be around for the next few years while Vulkan ramps up and develops an ecosystem. For these reasons, you'd be foolish to rule out learning GL entirely. Even so, you'll have a much easier time with it, and become a better GL programmer (and a better GPU programmer in general), if you start off with something that helps you to understand the GPU, instead of understanding how they worked 20 years ago.

Of course, there's one option more. I don't know whether this is relevant to you in particular, but I feel I should say it for the other visitors anyway.

To be an indie games developer, or a game designer, or to make VR experiences, you don't need to learn Vulkan or GL. Many people get started with a games engine (Unity or UE4 are popular at the moment). Using an engine like that will let you focus on the experience you want to create, instead of the technology behind it. It will hide the differences between GL and Vulkan from you, and you don't need to worry about which is supported on your platform. It'll let you learn about 3D co-ordinates, transforms, lighting, and animation without having to deal with all the gritty details at once. Some game or VR studios only work in an engine, and they don't have a full-time GL expert at all. Even in larger studios which write their own engines, the people who do the graphics programming are a minority, and most of the developers work on higher-level code.

Learning about the details of how to interact with a GPU is certainly a useful skill, and one that many employers will value, but you shouldn't feel like you have to learn that to get into 3D programming; and even if you know it, it won't necessarily be something you use every day.


What is the simplest way to compute principal curvature for a mesh triangle? I have a mesh and in the region around each triangle, I want to compute an estimate of the principal curvature directions. I have never done this sort of thing before and Wikipedia does not help a lot. Can you describe or point me to a simple algorithm that can help me compute this estimate?

Assume that I know positions and normals of all vertices.

When I needed an estimate of mesh curvature for a skin shader, the algorithm I ended up settling on was this:

First, I computed a scalar curvature for each edge in the mesh. If the edge has positions $p_1, p_2$ and normals $n_1, n_2$, then I estimated its curvature as:

$$\text{curvature} = \frac{(n_2 - n_1) \cdot (p_2 - p_1)}{|p_2 - p_1|^2}$$

This calculates the difference in normals, projected along the edge, as a fraction of the length of the edge. (See below for how I came up with this formula.)

Then, for each vertex I looked at the curvatures of all the edges touching it. In my case, I just wanted a scalar estimate of "average curvature", so I ended up taking the geometric mean of the absolute values of all the edge curvatures at each vertex. For your case, you might find the minimum and maximum curvatures, and take those edges to be the principal curvature directions (maybe orthonormalizing them with the vertex normal). That's a bit rough, but it might give you a good enough result for what you want to do.

The motivation for this formula is looking at what happens in 2D when applied to a circle:

Suppose you have a circle of radius $r$ (so its curvature is $1/r$), and you have two points on the circle, with their normals $n_1, n_2$. The positions of the points, relative to the circle's center, are going to be $p_1 = rn_1$ and $p_2 = rn_2$, due to the property that a circle or sphere's normals always point directly out from its center.

Therefore you can recover the radius as $r = |p_1| / |n_1|$ or $|p_2| / |n_2|$. But in general, the vertex positions won't be relative to the circle's center. We can work around this by subtracting the two:
$$\begin{aligned}
p_2 - p_1 &= rn_2 - rn_1 \\
&= r(n_2 - n_1) \\
r &= \frac{|p_2 - p_1|}{|n_2 - n_1|} \\
\text{curvature} = \frac{1}{r} &= \frac{|n_2 - n_1|}{|p_2 - p_1|}
\end{aligned}$$

The result is exact only for circles and spheres. However, we can extend it to make it a bit more "tolerant", and use it on arbitrary 3D meshes, and it seems to work reasonably well. We can make the formula more "tolerant" by first projecting the vector $n_2 - n_1$ onto the direction of the edge, $p_2 - p_1$. This allows for these two vectors not being exactly parallel (as they are in the circle case); we'll just project away any component that's not parallel. We can do this by dotting with the normalized edge vector:
$$\begin{aligned}
\text{curvature} &= \frac{(n_2 - n_1) \cdot \text{normalize}(p_2 - p_1)}{|p_2 - p_1|} \\
&= \frac{(n_2 - n_1) \cdot (p_2 - p_1)/|p_2 - p_1|}{|p_2 - p_1|} \\
&= \frac{(n_2 - n_1) \cdot (p_2 - p_1)}{|p_2 - p_1|^2}
\end{aligned}$$

Et voilà, there's the formula that appeared at the top of this answer. By the way, a nice side benefit of using the signed projection (the dot product) is that the formula then gives a signed curvature: positive for convex, and negative for concave surfaces.

Another approach I can imagine using, but haven't tried, would be to estimate the second fundamental form of the surface at each vertex. This could be done by setting up a tangent basis at the vertex, then converting all neighboring vertices into that tangent space, and using least-squares to find the best-fit 2FF matrix. Then the principal curvature directions would be the eigenvectors of that matrix. This seems interesting as it could let you find curvature directions "implied" by the neighboring vertices without any edges explicitly pointing in those directions, but on the other hand is a lot more code, more computation, and perhaps less numerically robust.

A paper that takes this approach is Rusinkiewicz, "Estimating Curvatures and Their Derivatives on Triangle Meshes". It works by estimating the best-fit 2FF matrix per triangle, then averaging the matrices per-vertex (similar to how smooth normals are calculated).


What technique would I use to create fake 3d walls as seen in Realm of the Mad God I wanted to make a little demo for a console with Realm of the Mad God esc graphics to brush up my C and SDL skills.

One problem I'm running in to is creating the 3D walls.
The console I'm working with does not actually allow for 3D hardware acceleration at this moment of time and thus I can't really use real 3D.
I noticed the perspective looked off in RotMG as well so I figured they must be using fake 3D also.

Now I have tried layering 2d textures on top of eachother as detailed here although this didn't quite work out so I modified the algorithm a bit and the blocks looked a lot better. Problem is they wouldn't sit next to eachother convincingly so I figured this is not the right tool for the job.

What type of algorithm should I use to recreate the 3D walls without actually using GL or other real 3D technologies?

What they're using in this game is called an orthographic projection. It is a special type of a projection, which does not bend objects, it does not mean that they are not using 3D.

Now, of course you can achieve this kind of effect using 2D coordinates only if you want. But it requires some mathematical understanding of the situation, as you need to do 3D calculations in order to calculate the correct 2D coordinates to use. If you do it this way, you can draw points or lines between the points after calculating the positions, and it  will give the effect you want.

But here in RotMG, they are also doing texture mapping on those walls, which uses information about the actual size of the surface in 3D space. This information is lost if you simply calculate the 2D positions of the corners without also doing the texture mapping yourself.

So here you have two options:

You can give up your console and start doing some real 3D stuff (OpenGL, etc...).
You can become very good at linear algebra, preferably including homogenous coordinates, project the points from 3D to 2D space, draw your lines/points in your console, or interpolate the 3D positions with some form of rasterization, if you want to decorate the wall using texture mapping. But once you understand all of that, I'm afraid you won't want to use your limited console anymore.


Ray Transformation to Object Space for Motion Blur My raytracer supports a wide variety of objects.  To intersect them, I use the standard technique of transforming rays into object-space.  This works fantastically until I add motion blur.

I model motion blur as a sequence of transforms (to simplify discussion, let's say exactly two) instead of one.  My approach is to take the inverse transform of the ray at both keyframes and lerp the positions/directions.

This seems to work fine for translations, but it breaks down for rotations.  E.g. here are two triangles undergoing 30 and 90 degree rotations:

(4 samples, MN reconstruction, the red samples came from near the two keyframes)

At the corners, I would expect the lerped samples to lie on a straight line between the the two vertices.  Instead, they bulge outward.  This is wrong.  In more interesting scenes with more interesting transformations, it causes a variety of failure modes.  E.g. here's a propeller undergoing a 45 rotation:

(100 samples, normals visualized)

Some problems are due to the BVH breaking (it assumes the extrema of objects lie at keyframes), but even a brute force render is incorrect.

I can fix all this by doing forward transforms only (transform object, not the ray), but this only works for objects where that is possible (only triangles, really).

How can I make my raytracer produce linear approximations to transformation (especially rotation) by transforming rays, not objects?

Lerping the ray positions/directions between keyframes should be equivalent to lerping the inverse matrices between keyframes and transforming by the lerped matrix. Trouble is, if the keyframes have different rotations, that lerped matrix will in general be something "weird", with shearing, nonuniform scale, etc.

I wouldn't be surprised if some of your intersection and shading routines don't work properly in such a distorted coordinate system, unless you've specifically tested and hardened them against such cases. (For example, taking the dot product of two unit vectors doesn't give the same answer in a sheared coordinate system as in an orthonormal one.)

This is just a guess, but it might work better if you choose an interpolation method where the translation, rotation, and scale (if applicable) are lerped separately (using quaternions for the rotation part) and re-combined.


How enlarge parts of a video like snapchat lens filter I writing the this video process app which requires the each video frame to be processed such that parts of the video are enlarged. As a reference, one of the snapchat lens filter enlarges the eyes of a person in the video. This is the exact effect I want to achieve.

How can I achieve this effect?

Update:

I have attached an example of eye enlargement from the web.

The answer is surprisingly simple: You move the vertices of the face-geometry.

To elaborate, the white wireframe mesh on the left of your image is placed so the geometry aligns with the photo. It is then texture-mapped with the photo, meaning that for each vertex in the mesh, a texture-coordinate is assigned (the point on the photo that should map to the vertex).

With every vertex assigned a texture-coordinate you can deform the mesh, doing ordinary texture-sampling at the interpolated texture-coordinates to have the image stretch - as if it was painted onto the head.

If you need to do this in realtime, you can use your graphics-card to do this; texture mapping is one of the things GPUs are really good at.

Now, if you want to do this on a video, you need to track the facial features to continuously align the geometry to the video. This is not an easy task, but you can probably find a wealth of libraries to help you out.

I am a bit uncertain which of the above would be difficult, so please feel free to ask any questions.


Is there a objective reason for matrix naming conventions? I'm probably going to screw this up but.... In all the graphics libraries I've seen various matrices are often called something along the lines of

viewProjection
modelViewProjection
worldViewProjection
mvp
wvp

etc.

But, that seems unintuitive given that GLSL, HLSL, and most math libraries seem to multiply in the opposite direction.

In other words

viewProjection = projection * view;  // correct but unintuitive

vs

viewProjection = view * projection;  // incorrect but naming matches order

It gets worse if you add in transpose or inverse since now the order of operations vs the order of parts of the name of the matrix make zero sense whatsoever. a worldViewProjectionInverseTranspose would be

transpose(inverse(projection * view * world))

Well you can read it backward but as far as order of operations and position in the expression its

5 ( 4 ( 1 * 2 * 3 ) )

Is there any objective reason for the names to be backward from the operations that construct them or is it just some naming convention that stuck from some example lost in obscurity?

I'm asking this question because being that it's unintuitive I find it's always hard to remember the order. If worldViewProjection equaled world * view * projection it would be easy since the name matches the order. I'm hoping there's some other objective reason that might help remember vs just remembering always multiply backward from the order of the parts of the commonly used names.

I think the naming order is intuitive because it is in reading order (left to right), e.g., worldViewProjection means that your point/direction is first multiplied by the world matrix, then the view matrix, and then the projection matrix. In this manner, you know the correct multiplication order by just reading the variable name and you do not have to think about mathematics at all. In your case, I think the problem is that your intuition seems to be focused on how you write the equations on paper or code.

The idea that a point is a column vector and matrices are multiplied on the left is also just a mathematical convention that you are used. You can use a row vector and multiply on the right by the transpose matrix, and the math is exactly the same, but would imply a reverse writing order in compound transformations. Some people just prefer one notation over the other.

I have read some old computer graphics books that use row vectors as points/directions, but it seems that most modern books have converged on using column vectors. IMHO, maybe this change during the years is one of the sources of confusion for so many new CG students.

For instance, if you choose to represent points as row vectors you would write something like this in your code: new_point = point * world * view * projection. On the other hand, if you use column vectors you would write this: new_point = projection * view * world * point. The important thing is that these matrices must not be the same, that is, the world matrix in the first code is the transpose of the world matrix in the second code, and the same is true for the others. As you can see, if you choose one math convention over the other, this will probably have influence on your naming conventions while writing your code.


Is it expected that a naive path tracer takes many, many samples to converge? I have a naive diffuse/emissive material path tracer implementation (pinhole camera) and am seeing it take a very large number of samples per pixel to converge - like probably in the millions - and was wondering if that was normal for "naive" path tracer implementations?

Here is 10,000 samples per pixel:

And here's a 10,000spp image using cosine weighted hemisphere samples as comparison:

For each pixel I do this:

Cast a ray out into the scene for the pixel (starting at near plane, of the direction of the camera to the pixel's location on the near plane).
When an object is hit, I calculate a uniform random direction in the hemisphere defined by the normal and continue.
At each level, i return emissive + (2 * Dot(normal, bounceRayDir) * ColorFromBounceRay * DiffuseColor/pi)
If nothing is hit by a bounced ray, i return black.
I'm working with 3 floating point color channels, where the intent is that 0-1 corresponds to 0-255 in the final output uint8 per color channel.
I allow 5 bounces max.

After averaging all the samples for the pixel, I do this to convert it from HDR to LDR:

For each color channel i do gamma correction: channel = powf(channel, 1.0f /2.2f)
I then multiply the channel value by 255, clamp it from 0-255 and cast it to uint8 for the final pixel color.

The source code can be seen here in the gist below.  It includes windows.h for bitmap headers, but otherwise has no non standard headers, and no library dependencies:
https://gist.github.com/anonymous/0dc9f7f5abf15f8fdb9aa84ecfcf67d5

Without explicit light sampling, I'd expect very slow convergence.

Any path that doesn't happen to hit the light source before being terminated will be useless (contributes zero). Since you have a small light source, the vast majority of paths will not hit it. It would be interesting to add some statistics to your tracer to see how many paths return zero vs nonzero. I'd guess you'll find something like 99–99.9% of them are zero, so you're only getting one useful sample out of every 100–1000 or so paths traced.

Naive path tracing is more reasonable with e.g. a sky light, which almost every path can be expected to hit.


OpenGL ES 3 - Uniform buffer object with float array I'm trying to load an array of float to a fragment shader using a uniform buffer object, but it doesn't work.
In the fragment shader I declared the following uniform block:

/// Spectrum samples.
const int samples = 31;

layout(std140) uniform SpectralDataBlock {

    float illuminant[samples];
    float object[samples];
};

Then on client side in my C++ code:

float illuminant[31] = {
    82.754900,91.486000,93.431800,86.682300,104.865000,117.008000,117.812000,114.861000,115.923000,108.811000,109.354000,
    107.802000,104.790000,107.689000,104.405000,104.046000,100.000000,96.334200,95.788000,88.685600,90.006200,89.599100,
    87.698700,83.288600,83.699200,80.026800,80.214600,82.277800,78.284200,69.721300,71.609100
};

float object[31] = {
    0.051, 0.05, 0.049, 0.049, 0.049, 0.049, 0.048, 0.047, 0.045, 0.044,
    0.044, 0.044, 0.044, 0.044, 0.045, 0.047, 0.05, 0.057, 0.072, 0.109,
    0.192, 0.332, 0.486, 0.598, 0.654, 0.686, 0.7, 0.707, 0.718, 0.724,
    0.729
};

std::copy(illuminant, illuminant + 31, spectralData);
std::copy(object, object + 31, spectralData + 31);

blockId = glGetUniformBlockIndex(program, "SpectralDataBlock");
glUniformBlockBinding(program, blockId, bindingPoint);
glGetActiveUniformBlockiv(program, blockId, GL_UNIFORM_BLOCK_DATA_SIZE, &blockSize);

glGenBuffers(1, &bufferId);
glBindBuffer(GL_UNIFORM_BUFFER, bufferId);
glBufferData(GL_UNIFORM_BUFFER, blockSize, spectralData, GL_DYNAMIC_DRAW);
glBindBufferBase(GL_UNIFORM_BUFFER, bindingPoint, bufferId);

Unfortunately, the data doesn't get correctly loaded.

Are there any limitation in OpenGL ES the prevent to use float[] in an uniform buffer object? Are there any error in my code above?

Let's review what the specification says about std140 layout of arrays:

  If the member is an array of scalars or vectors, the base alignment and array stride are set to match the base alignment of a single array element, according to rules (1), (2), and (3), and rounded up to the base alignment of a vec4. The array may have padding at the end; the base offset of the member following the array is rounded up to the next multiple of the base alignment.

Emphasis added.

So, if you have an array of float, the array stride shall be the same as the base alignment of a vec4. Also, each array will be padded at the end to the base alignment of a vec4. Yes, a float[] takes up the same room as a vec4[] of the same size.

Welcome to the wonderful world of std140 layout.

If you want to have a real array of floats in a UBO... well, it's best to avoid wanting that. But if you have absolutely no other choice, then I suggest making it an array of vec4. Obviously, the size of this array will be the size you actually want, divided by 4 and rounded up:

const int samples = 31;
const int arraySize = ((samples + 3) / 4);

To fetch a single float from this array, use math:

arrayOfVec4[index / 4][index % 4]

Note that OpenGL ES 3.0 does allow non-constant array indexing of vec and mat types. It also allows the integer operations needed to pull this off.


How to dynamically calculate texture coordinates in a vertex shader with OpenGL? I modified my code, and tried to pass a set of integers specifying the face of a cube (top, bottom left...) to the vertex shader in order to draw multiple instances of a cube face.

I did this by trying to pass them as vertex attributes, but it isn't working as I expected, the value of the int should be between 0 and 5, but it's never a number between 0 and 5, in fact it seems to be much much larger than that.

This is my current shader:

#version 450

#define PI 3.1415926535897932384626433832795

layout(location = 0) in vec3 position; //(x,y,z) coordinates of a vertex
layout(location = 1) in vec3 norm; //a 3D vertex representing the normal to teh vertex
layout(location = 2) in vec2 texture_coordinate;
layout(location = 3) in int face_type;

out vec3 normal;
out vec3 vertexPos; //projected vertex
out vec2 texture_coord;

uniform mat4 model = mat4(1); //Position and orientation of the current object
uniform mat4 view = mat4(1); //Camera orientation and position
uniform mat4 proj = mat4(1); //the projection parameters (FOV, viewport dimensions)

//Taken from: https://gist.github.com/neilmendoza/4512992
mat4 rotationMatrix(vec3 axis, float angle)
{
    axis = normalize(axis);
    float s = sin(angle);
    float c = cos(angle);
    float oc = 1.0 - c;

    return mat4(oc * axis.x * axis.x + c,           oc * axis.x * axis.y - axis.z * s,  oc * axis.z * axis.x + axis.y * s,  0.0,
                oc * axis.x * axis.y + axis.z * s,  oc * axis.y * axis.y + c,           oc * axis.y * axis.z - axis.x * s,  0.0,
                oc * axis.z * axis.x - axis.y * s,  oc * axis.y * axis.z + axis.x * s,  oc * axis.z * axis.z + c,           0.0,
                0.0,                                0.0,                                0.0,                                1.0);
}

void main()
{
    mat4 rotation = mat4(1);
    switch(face_type)
    {
        case 0:
            rotation = mat4(1);
            break;
        case 1:
            rotation = rotationMatrix(vec3(0,0,1), -PI/2.f);
            break;
        case 2:
            rotation = rotationMatrix(vec3(0,0,1), PI/2.f);
            break;
        case 3:
            rotation = rotationMatrix(vec3(0,0,1), PI);
            break;
        case 4:
            rotation = rotationMatrix(vec3(1,0,0), PI/2.f);
            break;
        case 5:
            rotation = rotationMatrix(vec3(1,0,0), -PI/2.f);
            break;

        default:
            rotation = mat4(vec4(face_type,0,0,0),
                            vec4(0,1,0,0),
                            vec4(0,0,1,0),
                            vec4(0,0,0,1));
            break;
    }

    /*rotation = mat4(vec4(5,0,0,0),
                    vec4(0,5,0,0),
                    vec4(0,0,5,0),
                    vec4(0,0,0,1));*/

    vec4 projection = proj*view*model*rotation*vec4(position, 1.0);
    gl_Position = projection;

    normal = vec3(model*vec4(norm,1.0));
    vertexPos = vec3(model*vec4(position, 1.0)); //calculate the transformed pos
    texture_coord = texture_coordinate + vec2(face_type*(1/6.f),0);
}

And this is how I am passing the face information to the shader:

vector<Face> face_types = {Top, Bottom, Left,};
glEnableVertexAttribArray(3);
glBindBuffer(GL_ARRAY_BUFFER, experiment);
glVertexAttribPointer(3, 1, GL_INT, GL_FALSE, sizeof(Face), (void*)0);
glBufferData(GL_ARRAY_BUFFER, face_types.size()*sizeof(Face),
    face_types.data(), GL_DYNAMIC_DRAW);
glVertexAttribDivisor(3, 1);

To be able to give a good answer, we need to know what is the compiler error you are referring to.

At first sight your shader looks ok, although:

It always sets texture_coord.y to 0, which might not be what you want.
The .f notation (in orientation*(1/6.f)) doesn't exist in GLSL. It should be 1./6.. Moreover, I am not entirely sure if strict GLSL allows to multiply a uint with a float; a cast might be necessary.
You also need to make sure that on the receiving end, the fragment shader is declaring the texture coordinate as needed: in vec2 texture_coord;.

It is fairly common to set texture coordinates dynamically in the vertex shader, so you should be able to do so too.

Update to your additional questions and details:

  OpenGL throws an INVALID_OPERATION error

OpenGL error handling is quite tricky, since you have to poll for errors yourself, and it uses the same few error codes for wildly different functions. You have to identify exactly which function yielded the error, then refer to the documentation for the list of reasons this error can happen for that function. To make that task easier, the best way I have found so far is to always surround OpenGL calls with a macro doing an error check. You can find an implementation of a macro like that in the source code of bgfx for example.

  The error comes from my face type array being too big, but now I don't know how to pass the face type information into the shader, since I know I need 2048 elements at most

The layout declared in the shader must match the one declared through the OpenGL calls. The number of vertices is irrelevant on the shader side, but it must be consistent on the OpenGL declaration side. Unless you are using different indexes for the vertex attributes, you should have as many positions, normals, texture coordinates and face types.

The following snippet from your question seems to be declaring face types for only three vertices though:

glBindBuffer(GL_ARRAY_BUFFER, experiment);
glVertexAttribPointer(3, 1, GL_INT, GL_FALSE, sizeof(Face), (void*)0);
glBufferData(GL_ARRAY_BUFFER, face_types.size()*sizeof(Face), face_types.data(), GL_DYNAMIC_DRAW);
glVertexAttribDivisor(3, 1);

I am not familiar with glVertexAttribDivisor, but from what I read, it seems to be limited to instanced rendering.

You can find a simple example of binding in the examples of Dear ImGui.


What to reproject when doing temporal filtering in stochastic screen space reflection? I am in the middle of implementing stochastic screen space reflection based on frostbite presentation here. On page number 39, it is stated that we need to reproject the ray intersection location instead of the reflected surface location. I don't really get what's the difference between the two. Isn't ray intersection location the same as the reflected surface location. Can someone explain this?

And then in another ppt, they mention that

  I mentioned temporal reprojection, so let’s have a quick look at it
  now. The idea is simple: given a pixel’s 3D coordinate, check where it
  was in the previous frame, project it, and we have the texcoord to
  sample.

  … except when we move the viewport, reflections exhibit parallax. That
  is, the reflected objects move according to their depth, not the depth
  of the surface which reflects them. Attempting to reproject
  reflections using the closer depth results in smearing. So we
  calculate the average depth of the reflected objects, and reproject
  using that. This results in much reduced smearing.

Again I don't understand this at all.

Usual reprojection finds UV of current visible point into previous frame. For diffuse and rough objects it simply adds motion vector: vec2 UV_reprojected = UV + motion;
But for mirror-like surfaces it doesn't work, because we see not object (mirror) itself but what reflects in mirror (virtual object). The general idea to reproject reflected point is:

Find world space position for reflected point visible in reflector (Here we need to know reflected path length).
Substract world space delta between frames, so we found previous world position.
Project this point into reflector using previous camera position.
Project world space reflector point back to the screen.

The most hardest part is p.3, and is not easy analytically find this point, because reflector can change normal. Some interesting links about this topic:

Reflection reprojection in Uncharted. Here we assume that reflector is plane that changes transform, both camera and receiver are also dynamic.
Ray Tracing Gems chapter 32. (32.4.3) Theoretical approach that finds reflector position in terms of least squares. And interesting geometrical optic approach (here we assume that reflector point is locally sphere-like).


Maximum number of vertices after clipping a triangle against an AABB I clip a 3D triangle against a 3D Axis-Aligned Bounding Box (AABB) to obtain the largest planar polygon of the triangle contained in the AABB. My clipping algorithm is a (slightly modified) version of the robust (e.g. clipping planes have a small finite thickness) Sutherland-Hodgman algorithm as described in C. Ericson's Real-Time Collision Detection. I clip the triangle against each of the 6 planes constituting the AABB.

In order to avoid heap (de)allocation, I allocated a fixed size point buffer on the stack in advance for all the vertices of the obtained planar polygon. My question now is: what is the maximum number of vertices possible one can obtain after clipping a triangle against an AABB?

Based on the control flow, every examined vertex can result in two vertices during a polygon plane clipping. Thus $3*2^6$ vertices. Due to symmetry this becomes $3*2^3=24$ vertices. However, I always obtain less vertices in practice.

Funnily enough, I asked this exact question on Math.SE a couple years ago: Maximum number of vertices in intersection of triangle with box.

The answer is 9 vertices, because each of the 6 planes of the box can cut off one corner of the polygon, replacing one vertex with two. So 3 vertices + 6 added vertices due to clipping = 9 total.


Translate an arcball camera upwards or left-right This problem is giving me a really hard time.

I'm always looking at the origin and have a camera that moves around it in an arcball mode.

That part works beautifully:

I'm calculating the camera position as follows:

camera.pos.x = origin.x + distance (arrow) * cos(angleX)
camera.pos.z = origin.z + distance * sin(angleX)
camera.pos.y : Float = origin.y + distance * sin(angleY)

But now I also want to be able to shift the position of the camera laterally, i.e. on (or parallel to) the tangent plane to the sphere between the origin and the camera position. (That tangent plane being the screen plane.)

I.e. I want to shift the view matrix to the left/right and upwards/downwards, so that the camera is shifted laterally. (I need this for features such as moving the view on a touchscreen [two-finger gesture] or texture projection).

How should I go about this?

Here's a basic camera with panning (some parts removed for brevity). It's not arcball but should get the idea across. Basically it just transforms some directions of a plane in view space to one that is in world space.

rotate and pan are called on mouse move with the change in mouse position.

// Updates the view matrix based on parameters
void updateCam()
{
    // Calc eye position on sphere
    glm::vec3 eyePos = glm::vec3(
        rho * sin(phi) * cos(theta),
        rho * cos(phi),
        rho * sin(phi) * sin(theta));

    setEyePos(eyePos + focalPt);
    setFocalPt(focalPt);
    updateLookAt(); // Uses eyePos and focalPt to calculate a view matrix
}

// Input is a change in position
void rotate(glm::vec2 diff)
{
    theta += diff.x * rotateSpeed;
    phi -= diff.y * rotateSpeed;

    // Clamp to prevent user flipping camera at top of sphere
    phi = clamp(phi, 0.01f, 3.14f);
    updateCam();
}
void pan(glm::vec2 diff)
{
    // Up and right are the directions to pan when camera is facing forward
    glm::mat3 invView = glm::inverse(view);
    glm::vec3 dx = invView * glm::vec3(1.0f, 0.0f, 0.0f);
    glm::vec3 dy = invView * glm::vec3(0.0f, 1.0f, 0.0f);

    // Scale shift with scale so shift is relative to how far you are zoomed out
    focalPt += (dy * diff.y - dx * diff.x) * shiftSpeed;
    updateCam();
}


Is colorized subpixel rendering possible? Subpixel rendering is used most commonly to anti alias fonts. It works by leveraging the physical layout of the color components of a display to give geometry details to an image that are smaller than a pixel.

For instance, this font has a height of 3 pixels but can easily be read:

from: http://www.sitepoint.com/two-teeny-tiny-fonts/

This is a monochromatic case, and i could see this possibly working in a greyscale case, but it seems like it always has to be used in situations where you have high contrast.

Is it possible to use sub pixel rendering in a full color situation, such as those you'd find while doing a 3d render of a scene?

I think that it must not be, since it won't be high contrast enough, but does anyone have examples to the contrary?

Here's an example of a downsampling filter that takes pixel geometry into account: Increasing image resolution on portable displays by subpixel rendering.

Image (a) below is an image downsampled using pixel-based downsampling. Image (b) is downsampled using direct subpixel-based downsampling, which (as far as I can tell) effectively downsamples the R, G, and B planes of the image independently. Image (c) is downsampled using diagonal direct subpixel-based downsampling, which uses a diagonal pattern to improve apparent resolution in both horizontal and vertical directions.

You can see that the last two images look sharper, but also have the characteristic color fringing.

Here's the downsampling pattern used for the last image:


Help me grasp Anisotropic Filtering (AF) Lately I've been reading about texture filtering, namely Nearest-neighbor filtering, Bilinear filtering, Trilinear filtering, Anisotropic filtering, MIP maps, RIP Maps and so on.

From a high-level perspective I think I can understand these techniques, how the work and why they exist, with the exception of Anisotropic filtering. Anisotropic filtering is driving me nuts.

I can see the problem of having to texturize a surface that is at an angle relative to the camera, but I don't get how sampling trapezoidal footprints could solve this (although I can see the result). This is probably because I DO NOT understand how the trapezoidal footprint is calculated and how the enclosed telexes are weighted to sample the texture.

This article by Nvidia confuses me even more by using sentences as "when a texel is trapezoidal" or "anisotropic filtering scales either the height or width of a mipmap by a ratio relative to the perspective distortion of the texture". Trapezoidal texel? Scaling a MIPmap? What does this even mean?

Could you help me grasp how AF and AF levels work?

Please note that my goal is NOT to have an OpenGL or DirectX AF implementation, but rather to grasp how AF works from a high-level perspective.

To understand the nature of anisotropic filtering, you need to have a firm understanding of what texture mapping really means.

The term "texture mapping" means to assign positions on an object to locations in a texture. This permits the rasterizer/shader to, for each position on the object, fetch the corresponding data from the texture. The traditional method for doing this is to assign each vertex on an object a texture coordinate, which directly maps that position to a location in the texture. The rasterizer will interpolate this texture coordinate across the faces of the various triangles to produce the texture coordinate used to fetch the color from the texture.

Now, let's think about the process of rasterization. How does that work? It takes a triangle and breaks it up into pixel-sized blocks which we will call "fragments". Now, these pixel-sized blocks are pixel-sized relative to the screen.

But these fragments are not pixel-sized relative to the texture. Imagine if our rasterizer generated a texture coordinate for each corner of the fragment. Now imagine drawing those 4 corners, not in screen space, but in texture space. What shape would this be?

Well, that depends on the texture coordinates. That is, it depends on how the texture is mapped to the polygon. For any particular fragment, it might be an axis-aligned square. It might be a non-axis-aligned square. It might be a rectangle. It might be a trapezoid. It might be pretty much any four-sided figure (or at least, convex ones).

If you were doing texture accessing correctly, the way to get the texture color for a fragment would be to figure out what this rectangle is. Then fetch every texel from the texture within that rectangle (using coverage to scale colors that are on the border). Then average them all together. That would be perfect texture mapping.

It would also be exceedingly slow.

In the interest of performance, we instead try to approximate the real answer. We base things on one texture coordinate, rather than the 4 that cover the entire fragment's area in texel space.

Mipmap-based filtering uses lower-resolution images. These images are basically a shortcut for the perfect method, by pre-computing what large blocks of colors would look like when blended together. So when it selects a lower mipmap, it is using pre-computed values where each texel represents an area of the texture.

Anisotropic filtering works by approximating the perfect method (which can, and should, be coupled with mipmapping) through taking up to a fixed number of additional samples. But how does it figure out the area in texel space to fetch from, since it's still only given one texture coordinate?

Basically, it cheats. Because fragment shaders are executed in 2x2 neighboring blocks, it is possible to compute the derivative of any value in the fragment shader in screen-space X and Y. It then uses these derivatives, coupled with the actual texture coordinate, to compute an approximation of what the true fragment's texture footprint would be. And then it performs a number of samples within this area.

Here's a diagram to help explain it:

The black-and-white squares represent our texture. It's just a checkerboard of 2x2 white and black texels.

The orange dot is the texture coordinate for the fragment in question. The red outline is the fragment's footprint, which is centered on the texture coordinate.

The green boxes represent the texels that an anisotropic filtering implementation might access (the details of anisotropic filtering algorithms are platform specific, so I can only explain the general idea).

This particular diagram suggests that an implementation might access 4 texels. Oh yes, the green boxes cover 7 of them, but the green box in the center could fetch from a smaller mipmap, thus fetching the equivalent of 4 texels in one fetch. The implementation would of course weight the average for that fetch by 4 relative to the single texel ones.

If the anisotropic filtering limit was 2 rather than 4 (or higher), then the implementation would pick 2 of those samples to represent the fragment's footprint.


Performance impact during texture layer loading Recently I have been having a rather confusing performance drop with loading in layers of a 2D texture array, but only in one case from what I can tell, leaving me to speculate on why it is happening as I have never seen anything like this before.

Firstly to explain some context for this problem, I implemented a simple version of a virtual texture atlas based on some papers I read by NVIDIA and idTech which in short loads textures (which have been split into fixed size tiles) into various levels of the 2D texture array based on which layers are considered "free" in a CPU-side list of free indices. Sampling in the fragment shader is then preformed by accessing a page table stored as a persistently coherently mapped SSBO (for CPU-side updating) which associates offsets in the UVs with layers of the texture array.

Due to the fact that the texture when created is uninitialized, every layer is considered free, so texture tiles to be loaded will slowly fill up the uninitialized layers (preferring them over previously freed layers). Once all of the uninitialized layers have been used, additional tiles will then begin overwriting previously freed layers, if any exist when needed. Also note that I am using glCompressedTexSubImage3D to load in each layer and that I load in mipmap levels as well manually per layer with this call as well.

This system works fine and was tested rather extensively until I began to notice the problem which this question is about. In short, after all of the uninitialized layers of the texture array were used up, the next 5-15 (I don't know exactly how many, but it's just a few) texture tiles which are loaded suffer massive performance drops. After that, everything returns to normal and the problem never returns (from what I can tell). Also note that this is while some currently drawn textures were also in use at the lower end of the texture array which is right before where the first free tiles begin (due to the first in first out nature of the free list).

Upon examining the problem further, I noticed through using the NVIDIA Nsight graphics profiling tool that the call after glCompressedTexSubImage3D in the cases where performance was impacted blocks on the CPU for very long periods of time, usually 10-40ms per mipmap level (in this case it was glGetError, but any other call following it had the same effect after disabling error checking to ensure that was not the problem).

Since the API was blocking, I assumed that it was some sort of synchronization issue with the GPU perhaps due to the locality of the newly loaded tiles with respect to tiles currently being drawn as it may have considered them "in use". That hypothesis did not add up in my mind due to how long the calls were blocking for as each one was the length of multiple frames at 16ms as well as how this problem never appears to return even after multiple times through this same process, though I'm not ignoring that possibility.

Another odd thing I found was that during the time the next call was blocking, many tiny GPU commands were being executed according to NVIDIA Nsight, which I usually only see under any sort of drawing activity.

At this point I'm pretty much stumped and don't know what could possibly be causing this, so any help would be greatly appreciated. Additional details are provided below (and more can be requested if needed):

Texture Array (there are 3 of them):

Format: COMPRESSED_RGB_BPTC_UNSIGNED_FLOAT, COMPRESSED_RED_RGTC1,
COMPRESSED_RG_RGTC2
Extents: 256x256x512, 9 mipmap levels

GPU:

GeForce GTX 960M

NVIDIA Nsight Results:

Wide view of 3 long frames where the slow texture loading happens (the line down the screen is centered on the first texture loading call):

Zoomed in view near the very start of the first slow call to glCompressedTexSubImage3D (which cannot be seen as it only blocks for a few microseconds), note the many tiny green lines below the blocking glGetError calls representing GPU commands (as again the line down the screen is centered on the first texture loading call):

From the way the problem only manifests when you start overwriting previously used texture tiles, I suspect you're running afoul of some driver heuristic that's attempting to manage the lifetime of the texture data on the GPU.

In general, when you update buffer or texture data from the CPU, the driver has to make sure it doesn't overwrite anything that might still be in use by the GPU (considering that the CPU and GPU run asynchronously to each other). There are various ways the driver can do this, such as internally double-buffering the texture, inserting sync points (for the CPU to wait on the GPU), or initially copying data to a temporary staging buffer and then later re-copying it to its final destination. All this stuff is under-the-hood and the driver makes its own decisions about what strategy to employ in any given situation, using some heuristics based on texture size, usage patterns, etc.

I can only guess at the details, but I suspect what's happening is that the driver is getting into a state where it thinks those previously-used texture tiles are still in use on the GPU somehow. Then when you start to overwrite them, it decides that it needs to make a second instance of the whole texture, and laboriously copies everything over before finally doing the updates you wanted—or something like that. That could be the many tiny draws you see in Nsight.

It's still a mystery why it's taking tens of milliseconds to do that, though. Maybe the textures are large enough that a second copy of them causes you to run out of VRAM? In that case, you'd get a lot of thrashing as things are spilled to main RAM. That's just a guess, though. As for why the bad behavior only happens for a few tiles and then goes back to normal, it could be that the driver heuristics are deciding to switch to a different strategy at that point.

Anyway, as for how to fix it, you might try using glInvalidateTexSubImage whenever you free a texture tile. This lets the driver know that you don't care what was in that texture tile before, which hopefully will convince it to just update the texture in-place rather than doing anything crazy with it.

Another thing to try is to make sure you don't try to re-use a texture tile in the very next frame after you free it, but give it a frame or two of "downtime" before re-using it. But it sounds like that's probably the case already, given that you prefer unused tiles before you start re-using them.

If that doesn't help, I'd recommend sending an email to devsupport@nvidia.com. The OpenGL driver team there can likely take your Nsight captures and help diagnose the issue, and fix it if it's due to a driver bug.


Subpixel Rendering for a Ray Tracer In font rendering, it is common to use subpixel rendering.  The basic idea here is to break the pixel into its RGB components and then compute a value for each separately.  Since each component is smaller than the entire pixel, higher quality antialiasing is possible.

There is an obviously analogous way to do the same for a ray tracer.  You do reconstruction filtering on each subchannel separately.

I was surprised that, however, I couldn't really find reference to ray tracers doing this.  Especially if you're already doing spectral rendering, this seems like an obvious thing to do.  There's this paper from a journal I've never heard of which seems to be related.  But overall, subpixel rendering just doesn't seem to be a common thing to do.  My question: why not?

This is perfectly possible

Although the difference may not especially noticeable, I would expect sampling taking into account the exact pixel geometry to give a slightly more accurate image. You just need to offset your pixel centres per colour component according to the (average) location of the subpixel(s) of that colour. Note that not all pixel layouts have a one to one correspondence between pixels and sub pixels.

For example, penTile RGBG has twice as many green subpixels as red and blue, as this image from Wikipedia shows:

I am not aware of any technical reason that would prevent this being used to produce arbitrary full colour images. In fact a colourful scene will have less noticeable colour artefacts than black on white text, which makes the colour differences hardest to camouflage.

Fonts are rendered on demand

The relevant difference between rendering a raytraced scene and rendering fonts is that fonts tend to be rendered on demand, and can take into account the screen being used. Contrasting with this, a raytraced scene is often prerendered and then displayed on many different types of screen (with different pixel geometry). For example, displaying your raytraced image on a webpage will prevent tailoring it to a specific monitor type.

If you were designing a realtime raytracing program and you had access to check the pixel geometry of the monitor, then you could raytrace to the specific sub pixel layout. However, offline raytracing that produces a still image can only be tailored to a single type of pixel geometry, which will then make the image look worse on any other pixel geometry. You could work around this by rendering a set of different images and choosing the appropriate one when it is later displayed on a particular type of monitor.

There is unlikely to be a long term benefit

So there is no reason you couldn't develop sub pixel rendering for a raytracer, but it means taking into account an aspect of the monitor that is not always known. Another thing to bear in mind is that you will be developing this software for a shrinking market. Sub pixel rendering is useful for screens that have relatively low resolution. As more and more screens (even mobile screens) are approaching such high resolution that the human eye cannot detect the difference made by sub pixel rendering, your work is likely to be more of theoretical interest than practical use.


What are some methods to render transparency in OpenGL Alpha blending can be turned on to make surfaces transparent, like so:

glDisable(GL_DEPTH_TEST); //or glDepthMask(GL_FALSE)? depth tests break blending
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);

But this only works if objects are rendered in back to front order. Otherwise things in the background appear in front of closer objects, like the floor in the image below. For particles and GUI elements sorting would be OK, but for triangle meshes it seems like it'd be too much effort and slow, as discussed here: https://www.opengl.org/wiki/Transparency_Sorting.

What are the common methods to deal with this? I know this is quite broad and am not after in-depth implementation details, just a brief description of some approaches and what might be involved.

A set of techniques to avoid explicit ordering go under the name of Order Independent Transparency (OIT for short).

There are lots of OIT techniques.

Historically one is Depth Peeling. In this approach you first render the front-most fragments/pixels, then you find the closest to the one found in the previous step and so forth, going on with as many "layer" as you need.
It is called depth peeling because at each pass you "peel" one layer of depth. All your layer can be then normally recombined from back to front.
To implement this algorithm you need to have a copy of the depth buffer.

Another set of techniques are the blendend OIT ones. One the most recent and interesting one is the Weighted Blended OIT proposed by McGuire and Bavoil. It basically apply a weighted sum for all the surfaces that occupies a given a fragment.  The weighting scheme they propose is based on camera-space Z (as an approximation to occlusion) and opacity.
The idea is that if you can reduce the problem to a weighted sum, you don't really care about ordering.

Other than the original paper, a great resource for implementation details and problems of Weighted Blended OIT is in Matt Pettineo's blog.  As you can read from his post this technique is not a silver bullet. The main problem is that the weighting scheme is central and it needs to be tuned according to your scene/content.
From his experiments, whilst the technique seems to work fine for relatively low and medium opacity, it is failing when opacity approaches 1 and so could not be used from materials where big part of the surface is opaque (he makes the example of foliage).

Again, all come down to how you tune your depth-weights and finding the ones that fit perfectly your use-cases is not necessarily trivial.

As for what is needed to for the Weighted Blended OIT, nothing more than two extra render targets. One that you fill with the premultiplied alpha color ( color * alpha) and alpha, both weighted accordingly. The other one for the weights only.


Efficient methods to rasterize a scene of connected triangles Assuming I wanted to rasterize a scene of triangles that are all connected that is similar to the image below, what would be an efficient way to do it on a GPU and a CPU? The triangles can be of any size. Names of any methods utilized or any advice is appreciated!

Thanks!

It's an interesting question, because the advice changes over time. Having said that:

On a GPU, by far the most efficient way, as Nathan Reed said in the comment, is to use the rasterisation hardware because that is that it is designed for.
On a CPU, one of the most efficient ways is to more-or-less mimic what rasterisation hardware does.

GPU vendors are understandably quiet about the specifics of their hardware design, but the basic idea hasn't changed for a few decades.

Conceptually split the screen into tiles, e.g. of 8 by 8 pixels.
Traverse the tiles which overlap with a triangle using some variant on Pineda's method.
For each tile, compute a bit mask to find the exact pixels (or samples, if you're subsampling) in the bucket which are inside the triangle, using half-plane calculations. In GPU speak, the samples in a tile which are inside the triangle are called a fragment.
Do whatever processing is needed on the fragment.

On a modern CPU, fragment processing (including the half-space calculations) can be made extremely efficient by using vector instructions.


Using depth texture to mask local pixels I have a scene with camera facing billboards drawn with depth write off so they appear on top of other objects. I would like to draw each sprite so that any  obscured pixels are drawn with a very low opacity. So that obscured portions of each sprite are visible but very faint.

Could anyone point me in the direction of a solution? I am using Threejs but I imagine the solution will be shader (glsl) based. I already have the depth texture for the scene available.

Sketchfab implements this with its annotations, which is the effect I would like to acheive.

Use the depth texture you mentioned, comparing it to the fragment's current depth to apply an alpha factor to your color. Remember to enable blending on the host. You can also upload the color texture and apply a more controlled blending on the shader.

The shader code involved would look like this:

#version 430 core

out vec4 finalColor;

...

in vec2 screenUV;

...

uniform sampler2D depthTexture;

...

void main()
{
    float texDepth = texture(depthTexture, screenUV).r;
    float alpha = texDepth < gl_FragDepth? 0.5 : 1.0;

    ...

    finalColor = vec4(..., alpha);

}


Does smooth lighting work with Gouraud shading on single triangles? I'm currently working on a project where a 3D model gets computed through
an isosurface algorithm. This algorithm outputs single triangles
with vertices and normals, but without indices. So therefore I have
a lot of duplicates which can't be avoided, because of a software
restriction.

I was asked to write smooth lighting for this model, by sending the proper normals to the visualisation software, which uses OpenGL and runs under Windows. What I tried was to set the normals for duplicated vertices to the same direction but this also results in unsmoothed lighting

as visible here:

Is it possible to have smooth Gouraud shading on a 3D Model
which consists of many single unconnected triangles?

After thinking about it for some days I came up with a proof sketch (which is hopefully correct) that it is possible to use Gouraud shading on disconnected triangles, if they share an edge with the same normals as in the picture. The simple idea of this proof is to check that there is no discontinuity between the intensity of the triangles by approaching from each side to the edge. So given two triangles $T_1: (P_1,P_2,P_3)$ and $T_2: (P_1', P_2', P_3')$ which share an edge and let's assume w.l.o.g. that this edges are constructed by $(P_2,P_3)$ and $(P_2', P_3')$. Because Gouraud shading only computes the light intensity at the vertex positions and interpolate in between them, it's save to assume that the intensity at $P_2$ is the same as on $P_2'$, same goes for $P_3$ and $P_3'$. Because it's also save to assume that the values inside the triangles are getting computed correctly so it's only necessary to show that there is no discontinuity between the triangles. To interpolate inside the triangle it's rational to use Barycentric coordinates so that the intensity at any point inside the triangle is given by
\begin{align*}
 I  &= aI_2   + bI_3   + (1 - (a+b)) I_1\\
 I' &= a'I_2' + b'I_3' + (1 - (a'+b')) I_1'
\end{align*}
By now going from $I_1$ respectively $I_1'$ to the edge, a sequence can be constructed to archive this, by setting $a_n = \frac{c}{n+1}, b_n=\frac{d}{n+1}, c + d = 1$ and analogously for $a_n', b_n'$. Then the intensity is
\begin{align*}
 I_n  &= \frac{c}{n+1} I_2  + \frac{d}{n+1} I_3  + (1 - \frac{c+d}{n+1}) I_1\\
 I_n' &= \frac{c'}{n+1}I_2' + \frac{d'}{n+1}I_3' + (1 - \frac{c'+d'}{n+1}) I_1'
\end{align*}
If one now approaches the edge with:
\begin{align*}
 &\lim\limits_{n\rightarrow 0^+} I_n = cI_2 + dI_3\\
 &\lim\limits_{n\rightarrow 0^+} I_n' = c'I_2 + d'I_3'
\end{align*}
and using the requirement that $I_n$ and $I_n'$ share the same position $P_n$ if they are on an edge, which can be constructed by
\begin{align*}
 P_n &= \frac{c}{n+1} P_2  + \frac{d}{n+1} P_3  + (1 - \frac{c+d}{n+1}) P_1\\
 P_n &= \frac{c'}{n+1}P_2' + \frac{d'}{n+1}P_3' + (1 - \frac{c'+d'}{n+1}) P_1'
\end{align*}
with the fact that $P_2 =P_2'$ and $P_3 = P_3'$ it follows:
\begin{align*}
 &\lim\limits_{n\rightarrow 0^+} P_n = cP_2 + dP_3\\
 &\lim\limits_{n\rightarrow 0^+} P_n' = c'P_2 + d'P_3
\end{align*}
and $d = 1 - c$
\begin{align*}
 &\lim\limits_{n\rightarrow 0^+} P_n = cP_2 + (1-c)P_3\\
 &\lim\limits_{n\rightarrow 0^+} P_n' = c'P_2 + (1-c')P_3
\end{align*}
this results in $c = c'$ and therefore this gives the final result:
\begin{align*}
 \lim\limits_{n\rightarrow 0^+} I_n = cI_2 + (1-c)I_3 = c'I_2' + (1-c')I_3' = \lim\limits_{n\rightarrow 0^+}I_n'
\end{align*}
Thereby is possible to use Gouraud shading over disconnected triangles as in my case, because the intensity on the edge is the same on both triangles.

What actually happened in my case was that the normals differs slightly (thank you @Rahul for the tip) which therefore results in this strange look. I fixed them by averaging the normals over a small radius to ensure
the proper set up of the normals.


Framebuffer not detecting image The command,

fb-test -f 0

generates and displays a test pattern on the screen.

But, the command,

cp /home/root/test.jpg /dev/fb0

copies test image into dev/fb0 file but fails to display anything on the screen?

How to resolve this issue?

Copying the bytes of a JPEG file to the framebuffer device doesn't display the image represented by the JPEG file. The file needs to be decoded by an application that understands the JPEG format, to get the actual image. Your system is working normally. You can use a command like display to decode the file and draw the image to the screen.


How can I implement a complex sinusoidal function? The following is the 2d complex sinusoidal function,

$u_0$ and $v_0$ represent Fundamental Frequencies in $X$ and $Y$ directions respectively.

How can I represent $j$ (imaginary number)?

Edit:

here is my use-case:  ...  ...  I need to implement Gabor Filter using both spatial and frequency domain equations. In the link ...  http://www.cs.utah.edu/~arul/report/node13.html  ..., you can see that there are several equations. (14) is the equation of Gabor Filter in spatial domain. (15) is the equation of Gabor Filter in frequency domain. Hence, my question.

One way to represent a complex-valued function in a bitmap is to use one color channel for the real component and one for the imaginary component. For example, rendering a complex plane wave (your equation) with R = real, G = imaginary looks like this (click for shadertoy):


Normal vector in Phong Illumination Model should be normalized? I'm studying Illumination subject these days and I was given a problem which asks to calculate the intensity using phong illumination model. I'm little bit confused whether I should calculate normal vector using after normalize it or as it is. Normal is given below. So if you can please tell me what type of vector I have to use there and if I have to use normalized vectors then it's value.

$N=(0,0,-12)$

Short answer: yes.

Longer answer: yes, because the vectors you’re using are meant to represent directions, not directions-and-distances. Think of it in terms of light: it doesn’t matter how far a photon’s traveled, whether it’s from the sun or from a lamp on your desk—once it arrives at a surface, it’s going to get reflected in exactly the same way.

From a mathematical standpoint, you’re making lighting calculations based on the angles between the incoming light, the view point, and the surface normal. The dot product is defined as the length of the two input vectors multiplied by each other and by the cosine of the angle between them. To get just the angle—which you need for the lighting model—you need to factor out the length of the two vectors, i.e. normalize them.


Adding vbos to a vao once it has been created Is it possible to add vbos to vbos after their creation? For example (pseudo-code):

glGenVertexArrays(1, &_vao);
glBindVertexArray(_vao);
// do init stuff
glBindVertexArray(0);

// Much much later in a galaxy far away
glBindVertexArray(_vao);
glGenBuffers(1, &_instance_vbo);
glBindBuffer(GL_ARRAY_BUFFER, _instance_vbo);
glBufferData(GL_ARRAY_BUFFER, sizeof(blee), blee, GL_STATIC_DRAW);
glEnableVertexAttribArray(_vpos_att);
glVertexAttribPointer(_vpos_att, 3, GL_FLOAT, GL_FALSE, 0, (void*)0);

// Draw

So far it hasn't really worked. Looking at the bound vbos to my vao (using GL_VERTEX_ATTRIB_ARRAY_BUFFER_BINDING) shows no buffers bound. Though that may be another problem altogether.

So after debugging some more, I've found that yes it is possible. You cam add buffers later on but they will need to point to their own attributes.

My issues came from thinking I could have multiple vbos contributing to a single attribute. That doesn't work. You need multiple vaos for that scenario.


Change particular color in an image Suppose i have a image and I want to convert the portion of image that is red to blue(or any other color). One way to do this is extract red part of image using HSV or other color values and convert red hsv value to corresponding blue. Is there any other methods(better) for this. What sort of algorithms do Photoshop color replacement tool use?

In order to replace a color with another color, you need some sort of distance metric between the colors and a function for calculating the blending based on that distance.

Finding a distance between 2 colors can be tricky. RGB is a bad color space to measure the distance between colors for perceptual uses. HSV or YCbCr are better.

Once you have the colors in a color space you like, you need to decide how to calculate the distance. I generally calculate the hue distance for starters. If the hue of the color I'm considering replacing is within +/- some threshold (possibly set by the user), then I check to see if the saturation and value are within a certain range, too.

Once I've found a color that's within the ranges specified, I calculate a blending amount. Usually this will be 100% if the color in the image exactly matches the color I want to replace, and will fall off to 0 at the thresholds of hue, saturation and brightness.

You then need to calculate the color you're going to use as a replacement color. It's useful to offset the hue, saturation and value (or Y, Cb, and Cr components) by the same amount as the color in the image differs from the replacement color.

So now you have the original color in the image, the color to replace it with, and a blending amount. You still need to figure out which blend mode you want to use. I would start with what Photoshop calls "Normal" or "Over". That would be a linear mix of the two. If you have colors A and B with a mix amount of 25%, it would be calculated like this:

newColor = A * .25 + B * (1.0 - .25);

Here are some relevant answers I gave to similar questions on Stack Overflow:

How to Change the Hue of a Texture with GLSL

Better Image Coloring Logic/Algorithm


Can someone explain this formula for parse RGB to HSL? Im trying to complain the algorithm to calculate the HSL value form a RGB color. I understand the algorithm for the Hue, but im lost with the Lightness and saturation. Someone can explain why?

There is the algorithm:

$$R'=R/255$$

$$G'=G/255$$

$$B'=B/255$$

$$C_{max}=\max\left(R',G',B'\right)$$

$$C_{min}=\min\left(R',G',B'\right)$$

$$\Delta=C_{max}-C_{min}$$

Hue Calculation:

$$H = \begin{cases}
0^\circ & ,\Delta=0\\
60^\circ \times \left(\frac{G'-B'}{\Delta}\mod 6\right) & ,C_{max}=R'\\
60^\circ \times \left(\frac{B'-R'}{\Delta}+2\right) & ,C_{max}=G'\\
60^\circ \times \left(\frac{R'-G'}{\Delta}+4\right) & ,C_{max}=B'
\end{cases}$$

Lightness calculation:

$$L=\frac{C_{max} + C_{min}}{2}$$

Saturation calculation:

$$S=\begin{cases}
0 & ,\Delta=0\\
\frac{\Delta}{1-|2L-1|} & ,\Delta <> 0
\end{cases}$$

First it converts the red, green, and blue values from the 8-bit unsigned integer range to floating point values between 0 and 1. It then figures out which component is the maximum, which is the minimum and the distance between the minimum and maximum.

If the min and max are equal, then the saturation is 0, and it makes no sense to calculate a hue. Here they are setting the hue to 0, but in reality, it's simply undefined. Assuming that no hue equals 0 hue can cause problem if you later manipulate the colors, so be careful with that.

If the min and max are not equal, then we can calculate a hue. The shape of the HSL color space is a double hexagonal cone. What the next part does is attempt to figure out which segment of the hexagonal cone the hue is in. If the maximum of the red, green, and blue, was the red channel, then it's in the area bounded by -60° to +60°. If green is the max, then the hue is between 60° and 180°, and if blue is the max, then the hue is between 180° and 300° (aka -60°).

The division in the hue calculation generates a value between -1 and 1. If red is the maximum component, then green - blue is either the middle value minus the lower value, or it's the lower value minus the middle value. And since the middle and lower value are never greater than the maximum, dividing by the delta will always give a value between -1 and 1. Once you have a value between -1 and 1, you multiply it by 60 to get the degrees. For the other 2 cases, you add either 2 or 4 before the multiply in order to rotate it 120° or 240° for the green and blue.

Next it calculates the lightness by simply averaging the min and max components. (This is a very poor way to represent the perceived brightness of a color, by the way.)

Finally, it calculates the saturation. If the delta is 0, then saturation is 0. Otherwise it divides the delta by 1 - abs(2 * L - 1). Essentially, it's saying that saturation is proportional to the distance between the minimum and the maximum values and inversely proportional to the lightness's distance from 50% lightness. In other words, as you move away from the vertical center of the double hex cone, towards the pointy ends, the distance between the min and max components has a bigger effect on saturation. A difference in the delta of, say, .1 at the center will not show much difference in saturation, but when the lightness is very close to either 0 or 1, it will make a large difference in saturation.


Does this type of memory-mapping for a display exist? Most information about memory mapped displays on the net are about those in which there is essentially a location in main memory for each pixel on the display. A hypothetical 1024 x 512 display would therefore have 524,288 locations each mapped to a unique pixel. To set a pixel to a particular color, all you basically need to do is write an RGB value into the corresponding memory location for that pixel, eg: MOV 0xFF3C0A, 0x00BB75. The first argument in that instruction is the address holding the RGB value and the second is the address of the pixel you are copying it to.

However, this can use up a quite a lot of main memory, which might not be ideal for certain systems. In a machine with a 32 bit or higher word size, it could instead be possible to have a single "display output register" into which you would write a single word of data containing the X and Y coordinates as well as the RGB value for the pixel you want to set. For a 1024 x 512 display with a 12-bit color depth, 31 bits would suffice to specify all this. The data 'written' into that register would either be sent to a display adapter or sent directly to the display and decoded there. This would save over half a megabyte of memory while still retaining the memory-mapped model.

So is this actually a thing or not?

In your 31 bit solution, the thing most likely to happen there is that you'd give an extra bit to green and make a 32 bit solution.  In 16 bit color for instance, r and b get 5 bits while g gets 6 bits, because your eye can distinguish more greens.

Or maybe you'd use that extra bit to have 1 bit alpha? Not sure if that'd be very useful though...

As far as the rest, whether you have a gpu or not, writing to single pixels at a time is bad form.

Instead, you ideally want to do block operations that either fill or copy large strips of memory.

For some things, like alpha blending, that isn't able to be done as a block operation very easily so is a more expensive operation in general.

As a solution to this, modern computers have a processor that is closer to the video memory, so there isn't too much latency when pushing pixels back and forth across the main bus to do things like calculate alpha blended values involving the source and destination pixel.

Also, instead of fewer more powerful cores, there are many less powerful cores, which helps address the situation of needing "one off" pixel operations.

That processor I'm referring to is the gpu and the programs you write for it are called shaders (;

What you are asking about doesn't exist to my knowledge and seems like a bad usage pattern, so seems unlikely to exist IMO.


How to ensure a fragment shader only runs once I am trying to programmatically figure out how many "texture2d " calls the gpu can make before the fps drops below 60fps.

My current way of doing this is to have a fragment shader with a for loop that will run a specified number of times based on a passed in uniform. So each frame the for loop will run more times, each time sampling the texture more times.

Then on the cpu I can monitor the fps and see when it drops.

My problem is if this shader runs on more then one pixel then the test is worth nothing.

How can I ensure from the cpu code and vertex shader that the fragment program only runs once?

I don't think your approach is viable for a variety of reasons.

a GPU is heavily cache-dependent for performance. Pulling the same pixels over and over will just measure how fast a cache hit is. Sampling random pixels will only measure a cache miss. Neither are useful statistics.
each texture2d call will pull in multiple pixels to leverage that, in a fragment shader, neighbouring fragments will want neighbouring pixels.
The driver could optimize your shader to pull out the loop if it sees that each iteration does the exact same computation or if the computed result is not used.


gl_FragCoord and glViewport I've been scanning the specifications for both OpenGL and GLSL, and I can't find a definitive answer as to how exactly are gl_FragCoord's x and y components computed, other than they are the

  fragment’s window-space position

(OpenGL 4.6 specification, page 491)

which does not help me understand why glViewport() doesn't seem to affect the values of gl_FragCoord in the fragment stage, since that is the case for window-space coordinates in the vertex stage.

The actual problem I'm trying to solve is rendering a bunch of draws (that depend on gl_FragCoord for their shading) to an atlas, without modifying the shaders of the draws to be aware of the atlas. Here's pseudocode for the driving part (all viewports have identical dimensions, just different offsets):

    for (size_t i = 0; i < FrameCount; ++i)
    {
        auto& Vp = Viewports[i];
        glViewport(Vp.X, Vp.Y, Vp.X + Vp.Width, Vp.Y + Vp.Height);
        BindConstants(Vp.Width, Vp.Height);
        glDrawArrays(GL_TRIANGLES, 0, Triangles);
    }

BindConstants() simply binds a uniform with the viewport's width & height. When trying to simply output the UVs with the following shader:

#version 330
uniform vec2 Resolution;

void main(void)
{
    vec2 UV = gl_FragCoord.xy / Resolution;
    gl_FragColor = vec4(UV, 0.0, 1.0);
}

I get the following image, which clearly shows that colour is being clipped to 1.0 beyond the first viewport in a 5x5 atlas:

I've also done additional simple tests that show that gl_FragCoord is based off of the entire frame buffer's origin, rather than what's specified via glViewport().

I have worked around this issue by rendering to a temporary FBO and blitting to the target frame buffer at the desired offset, but it's silly to go to such lengths for a dead-simple use-case like this.

Am I missing something? Is it possible at all without a temporary FBO, and, above all, without modifying the shaders?

since that is the case for window-space coordinates in the vertex stage.

There are no "window-space coordinates in the vertex stage". Or at least, not in the vertex shader.
Window-space is not relative to the viewport; it's relative to the window (or framebuffer, in this case). The viewport transform is used to convert from NDC-coordinates (which are relative to the viewport) to window coordinates (which are not).
If you need your fragment shader to have viewport-relative coordinates, then you need to either compute them in the fragment shader (by providing the viewport as a uniform) or pass NDC coordinates from the vertex shader as an in/out variable pair (just compute gl_Position.xy/gl_Position.w).


Am I missing something for ambient occlusion? I'm trying to implement ambient occlusion in Python 3 and I'm seeing shadows beneath my reflective spheres but they seem very faint. I'm not sure if that means I've missed something, or if I just have a false impression of how much shadow results from ambient occlusion.

The result looks like this:

This is modeled as spheres (rather than triangle meshes). The 4 yellowish mirror spheres are hovering just above a very large sphere to approximate a plane, and the whole scene is surrounded by a very large white emissive sphere that provides the ambient sky light.

For each pixel sample rays are chosen with a Gaussian distribution around the pixel centre and more samples are chosen until the variance is sufficiently low. Rays reflect in the single specular reflection direction from the mirror spheres, and in any direction from the hemisphere of possible directions when hitting the floor. All rays eventually hit the white sky sphere and the colour is determined based on the losses along the way.

Is there some vital step I'm overlooking?

It's a bit hard to tell from your image, but it does look a bit faint. When debugging these kinds of things, it's always useful to strip down your scene as much as possible to remove any unnecessary complexity from the picture. In your case, try only creating a single diffuse sphere that touches the ground in one point. Give the ground and the sphere an albedo of 1. If ambient occlusion is implemented correctly, the point where the sphere touches the ground should have a pixel value of 0; the further away from the sphere you get, the closer the pixel values should go to 1 (not above - make sure nothing gets clamped when you output the image). Make sure to gamma correct your images.

The one thing that sticks out from your description is

  Rays reflect [...] in any direction from the hemisphere of possible directions when hitting the floor

That by itself is fine, but you need to make sure to multiply the rays by the Lambertian BRDF (i.e. dot(normal, ray)/Pi). Even better is to directly sample from a cosine hemisphere, in which case all factors cancel out. This blog post has all the important info in one place.


How to estimate what GPU (2D/3D/none at all?) is necessary? I have to do some operations on 2D grayscale images. The "normal" stuff like moving, alpha blending and resizing, but also "3D like" operations: Perspective projection, rotation around any axis (x, y, z).
All this happens in an embedded system, where I can choose from microcontrollers with "2D GPU" (e.g. Vivante GC355), "3D GPU" (Vivante GC3000), "no GPU" (Arm A53).

Safe approach: Take the 3D GPU and use OpenGL operations (e.g. "Rotate") to do the jobs.

But how can I find out the minimum requirements?
In case of the "no GPU" approach it's quite easy to estimate the CPU cycles for each operation and extrapolate.
How to do this for a GPU? The specifications say things like "x million pixels per second" - but what can it do to these pixels in that second?
The tutorials I found so far say nothing about it.

There are really two things you need to know about your application:

How many pixels do you need to push per second?
How many cycles do you need to spend on each?

For the first, you need to sum the size of your output framebuffer and all the intermediates to count the number of pixels written in each frame. Make sure to include overdraw: if you draw a full-screen background, and then draw a foreground that covers half of the frame, you need to multiply by $1.5$. Then just divide that by your frame time (e.g. 33 ms if you need to reach 30 fps) to count the pixels per second.

To count cycles, you need to know what size of program will be run on each pixel. For a transformation or colour correction, it might just be a couple of multiply-adds. Somebody who would write the shader code (or the Neon code in the no-GPU case) should be able to work this out for you. It'll be easier if you have a prototype running on a desktop GPU already.

There are also two important properties of the hardware to consider. First is the pixel fill rate, which you've already seen. This tends to be limited by the memory bandwidth of the system-on-chip instead of by the CPU or GPU itself. If you're buying GPU IP and integrating it into the chip itself, you'll need to talk to the vendor's application support people to find out what your design can achieve. If you're buying a whole system-on-chip, just use the value from the spec sheet.

Second is your cycle budget. This depends on the GPU or FPU itself, not on the memory bandwidth, so it's easier to calculate. You just need to talk to the silicon vendor to determine this. For a GPU it'll be the number of cores times the clock speed divided by the instruction cycles per clock - but more realistically, look at a synthetic benchmark for this chip to see its megaflops.

Comparing your number of pixels per second with what the system can achieve is easy, but you also need to multiply this by the cycles per pixel you want to spend, to get the total number of cycles per second you need, and compare this with what the compute core offers.

Don't forget you also need to weigh the difference in BOM cost and software complexity against the risk and cost of a board respin, and multiply your estimate by some fudge factor appropriate to your project risk. We can't help you with that part here :-)


Full Monte-Carlo Volumetric Scattering I would like to add full monte-carlo volumetric scattering to my path tracer, but I'm having a hard time researching how to do it. Let me explain what I would like to do:

A ray enters a material, and we apply the BTDF, then after some distance, a volumetric scattering event happens, after which (in the isotropic case), the ray scatters in any direction in the sphere. This repeats until the ray exits the material with another BTDF.

My questions are as follows:

How do I choose the distance between scatter events? Intuition tells me there should be some kind of scatter pdf, which gives the probability to scatter after a certain distance?

Would this be correct?
Would the pdf be a linear function for isotropic materials?
Does this function have a name or something I can Google?

Would Beer-Lambert still apply between scatter events?

I would think not. Since Beer-Lambert is a simplification of actual scattering calculations.
Then again, perhaps Beer-Lambert is a calculation at the micro scale, and path tracing is on a macro scale.

What is the volumetric equivalent to a BSDF? It looks like I can use a phase function such as Henyey-Greenstein to determine the new direction, but what do I use for attenuation?
Lastly, what are some better Google phrases for Monte-Carlo volumetric scattering?

Searching volumetric scattering, or SSS, ends up giving papers, methods, and blog posts about the simplfications of full Monte-Carlo simulation (Dipole, in-scattering, out-scattering, diffusion, etc.)

First of all, a good reference for Monte Carlo path tracing in participating media is these course notes from Steve Marschner.

The way I like to think about volume scattering is that a photon traveling through a medium has a certain probability per unit length of interacting (getting scattered or absorbed). As long as it doesn't interact, it just goes in a straight line unimpeded and without losing energy. The greater the distance, the greater the probability that it interacts somewhere in that distance. The interaction probability per unit length is the coefficient $\sigma$ that you see in the equations. We usually have separate coefficients for scattering and absorption probabilities, so $\sigma = \sigma_s + \sigma_a$.

This probability per unit length is exactly the origin of the Beer-Lambert law. Slice a ray segment into infinitesimal intervals, treat each interval as an independent possible place to interact, then integrate along the ray; you get an exponential distribution (with rate parameter $\sigma$) for the probability of interaction as a function of distance.

So, to answer your questions directly:

You can technically choose the distance between events however you want, as long as you correctly weight the path for the probability that a photon can make it between two adjacent events without interacting with the medium. In other words, each path segment within the medium contributes a weight factor of $e^{-\sigma x}$, where $x$ is the length of the segment. (This is assuming a homogeneous medium, but see section 4.2 in the Marschner notes linked above for what to do if it's inhomogeneous.)

Given this, a usually good choice for the distance is to importance-sample it from the exponential distribution. In other words, you set $x = -(\ln \xi)/\sigma$ and then leave out the $e^{-\sigma x}$ factor from the path weight.

Then, to account for absorption, you can use Russian roulette to kill off a fraction $\sigma_a/\sigma$ of the paths at each event. This is particularly necessary for very large or infinite media (think atmospheric scattering) where the path could bounce around for an arbitrarily long time if it's not killed. If you're only dealing with small and not-too-dense media, then it might be better to just factor in a weight of $1 - \sigma_a/\sigma$ per event, rather than using Russian roulette.
No, if you follow the importance-sampling procedure just described, Beer-Lambert is already incorporated implicitly in the sampling, so you don't want to apply it to the path weights.
The volumetric equivalent to a BSDF is the combination of the scattering and absorption coefficients $\sigma_s, \sigma_a$ and the phase function. By convention, the coefficients control the overall balance of transmission, scattering, and absorption, while the phase function is always normalized.

You could do something like this for BSDFs too; you could factor out the overall albedo, and have the directional dependence always be normalized. It's mostly a matter of convention AFAICT.
Try "participating media" (that is, a volumetric "medium"—plural "media"—which "participates" in light transport), and "volumetric path tracing".