Trying to understand environmental cubemaps I am trying to implement a physically based renderer, in OpenGL. I was planning to implement the Cook-Torrance BRDF. I read about radiance cubemaps (for reflections), like the one discussed here, for example. I a bit confused however. My test environment is Sponza with some other models thrown in, where you can move the camera around freely. How would an environmental cubemap work with a scene like that, where there are arcades or similar architectural pieces, and other things in the way? Would there need be multiple cubemaps?

Your confusion comes from fact that in some basic tutorials like one you point cubemap which is used for applying lighting is global and static - it does not come from actual geometry but from infite sky box which moves with camera position so it is valid everywhere.

And there is nothing mentioned about second type - local cubemaps which are actually computed using local geometry and are valid only in point of their placement.

This can be approached using multiple local cubemaps and connecting them with local objects situated around them but what is problematic is that it generates seams between them especially for high frequency reflections, which you probably would like to test.

For now you can generate few local cubemaps - each one centered at one of your small objects and used on it. They would move with objects if you plan to change their position. They would take into account your local geometry and, if you render it, your global skybox or skydome. You can precompute these cubemaps each time you change the position of your objects or do it every frame.

If you want to make your whole sponza reflective with high frequency, that's harder problem but one idea is to voxelize whole sponza with different mipmap voxel levels and then ray trace it as in voxel cone tracing technique. Voxelization could be done as precomputation but it is generally memory intensive.


Is it possible to turn a 3d rotation matrix (4x4) into its component parts (rotation, scale, etc.)? To be more concrete, I'm working on an iOS app, and have a CATransform3D struct (basically a 4x4 transform array).

Is it possible to deduce all the different "operations" this matrix implies? Things like how much rotation, scale, etc. it implies?

You can decompose the matrix $\mathbf{M} = \mathbf{TRS}$ into basic transformations: translation, scaling, and rotation. Given this matrix:

$$\mathbf{M} = \begin{bmatrix}
a_{00} & a_{01} & a_{02} & a_{03}\\
a_{10} & a_{11} & a_{12} & a_{13}\\
a_{20} & a_{21} & a_{22} & a_{23}\\
0 & 0 & 0 & 1
\end{bmatrix}$$

You can decompose the translation by inspection using the last column $\mathbf{t} = (a_{03}, a_{13}, a_{23})$.

For scaling, we know that the first three columns of the matrix corresponds to the bases (axes). We can get the scale by the length/norm of these vectors, i.e. how much the bases were scaled. So the scale is $\mathbf{s} = (s_0, s_1, s_2)$ where:

$$\begin{matrix}
s_0 = \left \|(a_{00}, a_{10}, a_{20}) \right \|\\
s_1 = \left \|(a_{01}, a_{11}, a_{21}) \right \|\\
s_2 = \left \|(a_{02}, a_{12}, a_{22}) \right \|\\
\end{matrix}$$

Now you have the scale, you can get rid of it using the $3\times 3$ sub-matrix that corresponds to $\mathbf{RS}$ by multiplying the matrix with the inverse of the scale $\mathbf{S}^{-1}$ to get $\mathbf{R}$

$$\begin{align}
\mathbf{(RS)S}^{-1} &=
\begin{bmatrix}
a_{00} & a_{01} & a_{02}\\
a_{10} & a_{11} & a_{12}\\
a_{20} & a_{21} & a_{22}\\
\end{bmatrix}
\begin{bmatrix}
s_0 & 0 & 0\\
0 & s_1 & 0\\
0 & 0 & s_2
\end{bmatrix}^{-1} \\
&=
\begin{bmatrix}
a_{00} & a_{01} & a_{02}\\
a_{10} & a_{11} & a_{12}\\
a_{20} & a_{21} & a_{22}\\
\end{bmatrix}
\begin{bmatrix}
1/s_0 & 0 & 0\\
0 & 1/s_1 & 0\\
0 & 0 & 1/s_2
\end{bmatrix}
\end{align}$$

Thus ($\mathbf{(RS)S}^{-1} = \mathbf{RI} = \mathbf{R}$):

$$\mathbf{R} =
\begin{bmatrix}
a_{00}/s_0 & a_{01}/s_1 & a_{02}/s_2\\
a_{10}/s_0 & a_{11}/s_1 & a_{12}/s_2\\
a_{20}/s_0 & a_{21}/s_1 & a_{22}/s_2\\
\end{bmatrix}$$

This is the final rotation matrix. You can further decompose it using many ways. It is quit lengthy but you can search for decomposing a rotation matrix.

This method only gives an equivalent values in the form of translation, scaling and rotation (the original matrix maybe the result of other types of transformations). It may has problems with floating point precision with the rotation angles if you further used the decomposed angles, rounding errors may accumulate in the computations. You should not use it unless you did not construct the matrix yourself.

If you are the one who constructed the matrix and wanted the decomposition in order to be able to edit and display the translation, scale and rotation individually and independently, probabbly the cleanest why is to store the components of $\mathbf{t}$, $\mathbf{s}$ and $\mathbf{r}$ in a transform class individually as vectors (maybe quaternion for the rotation). Only when you need the transform matrix, construct a $\mathbf{TRS}$ matrix from these components (You can cache the matrix until some component is changed).


gl_ClipDistance vs gl_CullDistance Why are there very similar yet distinct arrays: gl_ClipDistance[] vs gl_CullDistance[]?
What would be some use cases where the need for two distinct entities become apparent?

The two are different because they do different things. Clipping and culling aren't the same thing at all. If an edge crosses a clip plane, the primitive has to be clipped to be within that plane. This means generating new primitives from the old one.
If any vertex of a primitive is outside of a cull plane, then the entire primitive is culled.
You use the cull planes when you want the VS to be able to cull out primitives based on some calculation. You use clip primitives when you want primitive rendering to be bounded to some non-clip-space region (though line and point width can violate those borders).


How to blend a segment of an image with another image I want to put a segment of an image to another image. But to make it realistic I applied poisson blending. But the output is not good at all.

Output without poisson blending. The kid was cropped from another image.

With poisson blending

I have no clue what else to try. I though of calculating average value of the image segment where the kid is placed and then scaling the kids color to that average. But it should not work properly if that image segment is very white or black etc.

Move the child down slightly so the right foot is behind the shoulder.
Reduce their size almost 50%.
Blend only the outline of the child with the background, at most a few pixels.
The men appear to be standing straight while the child appears to be leaned forward slightly - rotate the top of the inserted image backwards ~5°.
Reduce the brightness of the child slightly.

Doing the above should result in a realistic result.


Same specular lighting on both tris in a quad (for "low-poly" "flat shading" look)? Here's an example of the flat shaded squares I want. But my squares are made of two triangles, and I'm getting flat shading of those triangles.

In openGLES 3.1 (android), I'm turning off interpolation in the fragment shader, and using the provoking vertex (or using the centroid/average), to get the same "low-poly" lighting throughout a tri. The specular calculation is done in the vertex shader, so it's only done once.

I have a flat sea, a mesh of squares (two triangles per quad), with a specular reflection of the sun. But instead of each square being lit the same, the two triangles forming a square get different specular values.

I worked out it's because the tris are at different locations. Although the sun's rays are parallel (with the same angle of incidence at all locations), the angle between each tri and the viewer is slightly different, depending on where the tri is relative to the viewer.

  This diagram (Intro to Computer Graphics, David J. Eck) only shows the light from one point, for observers at different locations, but you can see that light from different points would produce a shifted cone, having a similar effect on an observer at one place.

A brute force solution would be to store a special "location" for each vertex of each tri, and set "location" to the same value for each of the 4 verices of the two tris forming the square.

But that's inefficient, and I'm using a STRIP for the mesh. I don't think this brute force solution work, because any per vertex data would be used by the 4 squares sharing that vertex, wouldn't they? Or can you have per tri data?

  EDIT I currently have per vertex normals - used in the specular calc - and only the normal of the provoking vertex is used for a tri. This works out as per tri, since only one tri is provoked by each vertex. I could give the same "location" to both tris in a quad this way. But it's still brute force - is there a better way?

Or, can you tell which tri of a quad you're in, so you can decide what it's "location" should be?

It seems so simple, but doesn't fit with openGL ES's approach... is there a way to do it? Note it's openGL ES 3.1 (mobile).

If I understand you correctly, the problem is that the provoking vertex isn't the same vertex between the two triangles in a quad, due to using triangle strips. You could switch to using an indexed triangle list, and set up indices in such a way that the provoking vertex is the same in each quad. For instance, if one quad is vertices {0, 1, 2, 3} (CCW) then you might write {0, 1, 2, 3, 0, 2} in your index buffer so that 2 is the provoking vertex for both triangles. This should still be pretty efficient.

Another possibility (more involved, but more flexible) is to use the built-in gl_VertexID variable in your vertex shader to figure out which quad you're in (eg int quadID = gl_VertexID / 4), and use an appropriate shading location for that quad—the centroid of its vertices, for instance. However, for this to work, I think you would have to "un-share" the vertices so that each quad has its own 4 unique vertices (the two triangles in a quad can still share). So you would have to use primitive restarting, or again switch to indexed non-strip draws. You could also have per-quad data in an SSBO that you manually fetch using your computed quad index.


Can I run polygon insetting on the surface of a mesh? I have a set of closed loops on the surface of a non-self-intersecting triangle mesh, and I need to create inset versions of them on the surface of the mesh. In other words, I want to generate a straight skeleton from them.

Within each single triangle of the mesh, this algorithm should behave just like a 2D straight skeleton.

I've found several tools for offsetting loops of edges in a mesh, or generating loops equidistant from a point on the mesh, but I need to be able to input arbitrary loops of my own and offset them.

I ran through various corner cases by hand on some graph paper, and it seems to me the algorithm described by Felkel and Obdrzalek should be perfectly applicable, except the wavefront may at times have to switch between contracting and expanding when it encounters geometries that give it more space.

This algorithm seems like it would be useful in several CAM applications, does anyone know of an existing implementation?

This is more of a long comment

Yes you can do this. The problem lies in how you measure the surface. I tried to do this manually by progressively extruding tubes along the edges, and finding their intersection with the surface. It would work but is a bit hard to compute. Obviously you may want to take more steps than i did for a more accurate solution.

Image 1 Straight skeleton (magenta)  and insets by finding the intersection of tubes along edges, and then the created edges on mesh (blue)
From this experiment I can conclude that are many complex cases to consider.


Calculating Material/Texture Placement from Bitmap to 3D Model everyone!
I am currently developing a program that converts a 3D model to 2D isometric pixel art. Part of my program extracts the texture from an FBX file (as FBX files can have textures built in), and then the texture will be used when converting to vox (which is then converted to a sprite sheet). So, I have a 3D model of a bed here:

And then I extract the texture programatically, which leaves us with a jpg:

But then, when converting the 3D model to vox, I need to get the color from the specific locations on the bed where the model is being converted to voxels, to apply to each voxel. To do this, my thoughts are that I'll calculate where on this texture sheet would map to my current point on the 3D model, and get the color at that pixel.
The trouble is calculating this. My question is, how do I calculate how a 2D texture sheet is mapped to a 3D model? Thanks in advance for your answers!

For each visible voxel side you need to iterate over all triangles and find if ray from voxel side normal intersects that particular triangle. Having intersection point you can then calculate barycentric coordinates of this point within intersected triangle. And with barycentric coordinates you can calculate UV of intersection point and therefore sample color from original texture. If you need more than one pixel per voxel side then you can cast more rays.


Issue with image load stores and compute shaders I am trying to read an input texture, modify it and then write to an output texture from a very simple compute shader.

For some reason I am not able to write to the output image from the compute shader.

The screen is black (even though I have a red clear color).

Here is my application code:

    // Compute shader
    (...Code to setup images)
    glTextureStorage2D(m_color_texture, 1, GL_RGB32F, m_src_fbo_size, m_src_fbo_size); // <- Yikes, wrong format!
    glTextureStorage2D(m_temp_texture, 1, GL_RGB32F, m_src_fbo_size, m_src_fbo_size); // <- Yikes, wrong format!
    m_compute_shader.use();
    glBindImageTexture(0, m_color_texture, 0, GL_FALSE, 0, GL_READ_ONLY, GL_RGBA32F);
    glBindImageTexture(1, m_temp_texture, 0, GL_FALSE, 0, GL_WRITE_ONLY, GL_RGBA32F);

    glDispatchCompute(m_info.window_width / 32, m_info.window_height / 32, 1);

    // Full screen quad
    glBindImageTexture(0, m_temp_texture, 0, GL_FALSE, 0, GL_READ_ONLY, GL_RGBA32F);
    glDisable(GL_DEPTH_TEST);
    m_full_screen_quad_shader.use();
    glViewport(0, 0, m_info.window_width, m_info.window_height);
    glBindVertexArray(m_full_screen_quad_vao);
    glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

And here is the compute shader:

#version 440 core

layout (local_size_x = 32, local_size_y = 32) in;

layout (binding = 0, rgba32f) readonly uniform image2D input_image;
layout (binding = 1, rgba32f) writeonly uniform image2D output_image;

void main()
{
    vec4 texel;
    ivec2 p = ivec2(gl_GlobalInvocationID.xy);

    texel = imageLoad(input_image, p);
    texel = vec4(1.0) - texel;
    imageStore(output_image, p, texel);
}

If I just swap m_temp_texture with m_color_texture (which is the input image )    in the call to glBindImageTexture(0, m_temp_texture, 0, GL_FALSE, 0, GL_READ_ONLY, GL_RGBA32F); then I see the image, so I know it is not a problem with the frag shader. (Here is is though anyway):

#version 440 core

layout (binding = 0) uniform sampler2D input_image;

layout (location = 0) out vec4 fragColor;

//uniform float focal_distance = 50.0;
//uniform float focal_depth = 30.0;

void main()
{
    vec4 color = texelFetch(input_image, ivec2(gl_FragCoord.xy), 0).rgba;
    fragColor = color;
}

Is there some reason why my compute shader is not being called? Even if I just write imageStore(output_image, p, vec4(1.0)); in the compute shader, the screen is all black.  I do not know what I am missing.

glGetError does not return anything.

EDIT:

I originally failed to include the code for the texture creation (the first two lines).  This affects the qustion. Please see the answer below.

Originally mistmatching image formats (GL_RGB32F instead of GL_RGBA32F) were used in the creation of textures and the call to glBindImageTexture.

This error was hard to track down because the format (without alpha) only mattered (in this case) in the compute shader.

Using GL_RGBA32Feverywhere fixed the issue.


Color transform algorithm (that also works on saturation and value/lightness)? To change particular colors in an image, generally you multiply the RGB by a transformation matrix. Image manipulation programs such as gimp & photoshop have a selective color change function that allows a user to convert pixels within a certain "distance" to the original color in some color space to a new color. Eg, changing #00f blue into #ffd pale yellow.

I'm trying to implement such a function in a <canvas>. What color space and distance definition do these algorithms typically use, and then how do they compute the transformation matrix?

Very often, the Lab color space is used, since it is close to how humans perceive color. As to which distance to apply: This wikipedia article mentions a Delta E distance of 1.0 for a just noticeable difference. There are a lot of different formulas resulting in different thresholds.


Perspective correct interpolation of normal values I'm currently writing a software rasterizer and I'm at the point where I can draw arbitrary triangles with vertex colors and perspective correct texture mapping. I do point and directional lighting by linearly interpolating the vertex normals and it works fine, only on big triangles you can notice that the lighting is somehow distorted when viewed from a very flat angle. However, this is only visible when you're particularly looking for it but I was wondering if there is a way to have perspective correction on the normals too.

For the texture coordinates and other stuff this is being done by dividing the value by W or Z. But if you divide a vector by a single number you only change its magnitude, which is useless, as I'm normalizing the vector in the end for the lighting computation. So, is there a way to do it perspective correct?

Perspective correct interpolation of normals works just as it does any color or coordinate or other linearly varying attribute. Each component of each varying vector is interpolated independently as if they were scalars, using exactly the same equation.


How to visualize and programmatically work with huge point cloud I have created a quite large point cloud with about 800M RGB points. The exported .ply file is about 12GB large.
The commercial software I used to create it can render the cloud very smoothly on my Ryzen 1800X/64GB/Geforce 1080Ti.

Now, I would like to work with this point cloud - programmatically put trajectories into it, combine multiple clouds, have more control over the point size ...
Expectedly, using PyntCloud in a Jupyter Notebook failed miserably :)

Before I start learning something like Unity, are there any better suggestions? Potree maybe?
Due to my lack of experience, I cannot judge yet where I would have which possibilities for working with the data.

Depends how 'deep' you want to go.

With Unity/Unreal doesn't have to be 'programatically' - they do have GUI, you can just load your data with few button clicks. Potentially you could parse/convert your data to readable format obj might be easiest with for eg. python.

So not sure what programming language you have in mind but as you are mentioning some python modules I guess python.

Then you could display it in many different ways.

OpenGL (there are many Python bindings over the internet, check them out and choose one that suits you best)
Matplot, pygraph and similar also have an ability to plot in 3D
explore pandas as it is a great python library to deal with this types of datas

And for eg. you could put it all together inside a QT


How can I raytrace a scene that does not fit into memory? If the scene to be raytraced cannot be stored in memory, then without adding more RAM to the machine it seems unrealistic to render it in a practical time span, due to the need to load different parts of the scene from disk potentially several times per pixel.

Is there any way around this? I'm trying to think of some way of performing a large number of the calculations involving a particular subset of the scene all at once, to reduce the number of times it needs to be loaded into memory. Is there any other way of improving the speed in such a case?

If the scene does not entirely fit into memory, you are entering the field of out-of-core rendering. There are essentially two approaches here: a) Generate your scene on-demand b) Load your scene on-demand

The former approach aligns well with most animation workflows, where models are heavily subdivided using e.g. Catmull-Clark and can become very memory-intensive, but the base meshes themselves easily fit into memory. Pixar have a few papers about this (e.g. Ray Differentials and Multiresolution Geometry Caching
for Distribution Ray Tracing in Complex Scenes), but the gist of it is that models are only subdivided when they are hit by a ray, and only subdivided as much as is reasonable for such a ray (e.g. diffuse interreflection need less accuracy than mirror reflections). The rest is handled by a geometry cache, which keeps the subdivided models in memory and hopefully makes the process efficient by a good eviction strategy.

As long as all your base meshes comfortably fit into memory, you can easily go out-of-core and render meshes at subdivision levels that would never fit into memory. The geometry cache also scales nicely with the amount of memory you have, allowing you to weigh RAM vs. render times. This was also used in Cars I believe.

The second approach is more general and does not rely on heavy use of subdivision. Instead, it relies on the fact that your scene was most likely made by an artist and already comes partitioned into reasonably small objects that fit into memory individually. The idea is then to keep two hierarchies (kD-tree or bounding volume hierarchy): A top-level hierarchy that only stores bounding boxes of the objects in your scene, and a low-level hierarchy that stores the actual geometry. There is one such low-level hierarchy for each object.

In this approach, you ideally already store a bounding box along with each object on disk. As the scene is loaded, you only build the top-level hierarchy initially, meaning you only have to look at the bounding boxes and not the geometry. You then start tracing rays and traverse them through the hierarchy. Whenever a ray hits a leaf node in the top-level hierarchy (i.e. it hits the bounding box of an object), that object is loaded into memory and its low-level hierarchy is built. The ray then continues down into tracing that object. Combined with an object cache that keeps as much of the low-level hierarchy in memory as possible, this can perform reasonably well.

The first benefit of such an approach is that objects that are never hit are never loaded, meaning that it automatically adapts to the visibility in your scene. The second benefit is that if you are tracing many rays, you don't have to load an object immediately as it is hit by a ray; instead, you can hold that ray and wait until enough rays have hit that object, amortizing the load over multiple ray hits.

You can also combine this approach with a ray sorting algorithm such as Sorted Deferred Shading for Production Path Tracing to avoid thrashing due to incoherent rays. The mentioned paper describes the architecture of Disney's Hyperion renderer, used for Big Hero 6 I believe, so it most likely can handle scenes at production scale.


How to combine rotation in 2 axis into one matrix I already know about the matrices I have to use in order to perform rotations. If I have to rotate in z-axis and then in x-axis, I would do it in 2 steps. My question is, is it possible to combine both rotations into a single matrix? I will appreciate your feedback.

(This answer is essentially the same as Stefan's but I wanted to add some detail about row and column vectors, and how to determine which you are using.)

Yes, this is possible, but the details depend on whether you represent your vectors as rows or columns.

Column vectors

If you are using column vectors, you will normally transform them by left-multiplying your matrices:

vector = mRotateZ * vector;
vector = mRotateX * vector;

Of course, you can also do this in one step:

vector = mRotateX * mRotateZ * vector;

But matrix multiplication is associative, which means it doesn't matter which multiplication is performed first:

A * B * C = (A * B) * C = A * (B * C)

So we can write

Matrix mRotate = mRotateX * mRotateZ;
vector = mRotate * vector;

We have now created a single matrix, which is equivalent to first rotating about Z and second about X. This generalises trivially for any number of transformations. Notice that transformations are applied from right to left.

Row vectors

If, on the other hand, you are using row vectors, you will normally right-multiply your matrices:

vector = vector * mRotateZ;
vector = vector * mRotateX;

Again, writing it in one step, we get

vector = vector * mRotateZ * mRotateX;

which can be rewritten as

Matrix mRotate = mRotateZ * mRotateX;
vector = vector * mRotate;

Notice that in this case, the transformations applied from left to right.


Visualize the output of a Trowbridge-Reitz Half Vector Sampling Function I am attempting to visualize the raw output from PBRT's half-vector sampling function, based on the trowbridge-reitz distribution.  I'm isolating only the distribution and the associated functions necessary to sample it.

The function is based on the standard GGX half vector sampling formula:

$$\theta_s = arctan(\frac{\alpha_g\sqrt{\xi_1}}{\sqrt{1-\xi_1}})$$
$$\phi_s = 2\pi\xi_2$$

(from Walter GGX: http://www.cs.cornell.edu/~srm/publications/EGSR07-btdf.pdf)

but the PBRT version has a lot more black magic related to importance sampling only directions that make significant contributions based on the incoming "wo" angle.  That one detail is making it hard for me to find published papers which specify how to add this directional importance sampling to the standard GGX sampling function.

Anyway, these angles come back from the sampling function as a vector 3 and spherical coordinates with $$Normalize([\theta_s,\phi_s, 1.0])$$ which should perturb the surface normal, creating the half vector to reflect around.

At this point I am only trying to make sense of the output from the "sample wh" function.  To visualize the results (raw "wh" output) I made a little graphics utility to output the results to a "normal map".  Here is the output:

And a live webgl example here that continually runs new random numbers through the output (as if monte carlo sampling were running): http://rdtests.ml3ds-test.com/microfacettesting.html

My expectation of the output would be a somewhat smooth gradient blend of the wh orientations as the x and y slopes cross into each other.  But what i'm seeing is an area between the 2 axes that doesn't want to blend well.  The random noise is good/expected, but you can see that inside the x/y blend areas, even the noise becomes fairly non-random.

Can anyone give me some tips on what the half vector sampling should graph like, or some pointers to papers/published work that I can use to locate test data, graphs, or other educational material that could help me figure this out?

So for anyone who was interested - after studying the source material by Walter, I became convinced the error had to be in the rotation of the slopes back into "normal shading space".  PBRT does this in a one-liner: "CosPhi(wiStretched) * slope_x - SinPhi(wiStretched) * slope_y;" etc etc.  I had split it out into separate sinTheta, sinPhi, sinPhiX steps.  One of those steps wasn't correct so the rotation wasn't unstreching correctly.

So the correct visualization for a GGX normal distribution of half vectors is this:


What physical properties are “lacking” to keep this 3D scene from looking like a real photograph? So I know that I'm basically asking to identify the main problem to be solved in creating realistic 3d Graphics, but as someone without much technical experience in the field, I'm curious that if these problems can be identified, what the issue is with implementing them programmatically.

The picture below from Hitman is what I would call "ultra-realistic". But, it still definitely looks like a 3d model. If you were to put a photograph of the same scene with a real actor and background, there would probably be some differences obvious enough to be able to point out which is which.

Why is that? < 100% accurate subsurface scattering? Is the lighting ever-so-slightly off? etc.

PS, sorry for tag choices. I am not familiar enough with this SE to know what to choose. Please edit better ones in, if you know any.

Few things, but usually this is what it takes to make the difference:

1- the material reflection at his head, he is bald, yet the diffuse texture shows color difference where the hair is, this means he has a shaved head, not a natural bald, this should translate in reflection, take a look at his head, top right (top left for the image), the reflection is very smooth, no bumps at all, there should be some indication in reflection to the tiny hairs that you can see in the texture map.

2- the wrinkles in his face need to be emphasized, he looks like some kind of a worrier, yet he has very smooth skin like a 15 year old has, I'm not talking about deep scars, I am talking about skin texture and bumpiness due to rough way of life, skin bumps, moles, a minor infection that will affect both coloration and bumpiness.

3- his mouth is off, the size and positioning, I know that many people has small mouths, and a relatively large distance between the nose and the mouth, but that drives the attention of the viewers, it feels wrong, even if it was a photo of a real person, you can spare this kind of dragging attention to extreme shapes if reality is what you're after.

4- This is a minor thing, but it had a role to play, the background and outfit, they look fake, this drives the viewers automatically and subconsciously to feel that this is not real, try to put more effort in these, making the outfit less uniform, and the background to have more realistic pattern, with more hue variation.

5- And last, the eyes, there's no reflection, yet the lighting is low and the image is small, so I am not sure if this is due to lighting or to the material itself.

That all been said, all these remarks are minor, the artwork looks awesome, I don't think a CG artwork should look like a photo, a perfectly realistic CG artwork that looks exactly like a photo, will become just another photo to the viewer.


OpenGL layered rendering with multiple additional clip planes per view I want to render a scene from multiple positions and orientations. I think the geometry shader is a good choice to not use the Vertex and tessellation shaders for each camera position again. So I want to do layered rendering. I already did that once, but this time I simultaneously want to use additional clip planes. Each camera position has its own clip planes, which should not be visible from other cameras.
My question: is it possible to define clipPlanes (glClipPlane) for a specific layer? Or is it better to do that during the fragment shader step, by doing dot-product, check the sign and discard if necessary?
Does I have other options (except: rendering the scene in a loop for each position / orientation)?
Thanks a lot!

The GS is responsible for assigning a particular primitive to a particular layer. The GS is also responsible for writing values for gl_ClipDistance for a particular primitive. So the GS is perfectly capable of doing both of those things: assigning a primitive to a layer and doing the clip-plane computations for that layer.
And if you need to "deactivate" a clip plane for a particular primitive assigned to a particular layer, just assign the gl_ClipDistance[i] for that to 1.0 for all of its vertices.


Controlling Sobel edge thickness I'm using a Sobel filter to draw an outline in a shader. How do I control its thickness? Here's my outlining code:

float3x3 Kx = {
-1, 0, 1,
-2, 0, 2,
-1, 0, 1};

float3x3 Ky = {
1, 2, 1,
0, 0, 0,
-1,-2,-1};

float Lx = 0;
float Ly = 0;

for (int y = -1; y <= 1; ++y) {
    for (int x = -1; x<= 1; ++x) {
        float2 offset = float2( x, y );
        float3 tex = tex2D(sampler, uv + offset).rgb;
        float luminance = dot(tex, float3(0.3, 0.59, 0.11));

        Lx += luminance * Kx[y+1][x+1];
        Ly += luminance * Ky[y+1][x+1];
    }
}
float color = sqrt(Lx*Lx + Ly*Ly);
return float4( color, color, color, color );

I've done this 2 different ways in the past:

Apply a pre-blur to the image before running the Sobel operator on it. This will have the side effect of getting rid of any noise that's smaller than the blur kernel, but it will also thicken the lines you end up drawing, too.
Apply a post-process where you thicken the lines by either stamping a shape at every point that's an edge, or using some sort of dilate process.


Confusion about how inverse bind pose is actually calculated and used? I am trying to do skeletal animation using Assimp and the inverse bind pose matrix just trips me up. I will give a little example to illustrate my point.

Root
    Bone A
        Bone B
            Bone C

Let's say we have a hierarchy like above. To get a vertex in root space into C space, I would have to do (CBA) * v. The inverse bind pose, which is supposed to get a vertex from bone space to root space, should be the inverse of this. Therefore, the way to calculate inverse bind matrix should be (CBA)^-1 or (A^-1)(B^-1)(C^-1). The formula then is (A^-1)(B^-1)(C^-1) * v, which makes sense. However, I am not able to reproduce the result of Assimp's mOffsetMatrix (inverse bind matrix caculated by Assimp) with (CBA)^-1. (ABC)^-1, however, surprisingly produces the correct result, albeit a little marginal error here and there which I attribute to floating point error.

To get a vertex in root space into C space, I would have to do (CBA) * v.

Well, yes. But that's not actually what you want to do in skeletal animation. You have it reverse.

It's the other way around. You assume you have a point in C space and want to compute its position in root/global space, in order to pump it further through the transformation pipeline, which gets increasingly more global with each transformation until your point ends up in eye-space (and further). That's why A doesn't transform from root space into A space, it transforms from A space into root space. So $ABC$ transforms from C space into root space.

Now your mesh vertices already are in root space, that's where the inverse bind pose comes into play, which transforms a point from root space into C space. So you basically transform your point from root space to C (or whatever joint) space with the inverse bind matrix that's only dependent on the joint and constant over the whole animation and then from that joint space back into root space with the animation-dependent joint transformation.

So C's transformation matrix for frame $t$, that transforms a vertex from unanimated root space, which is how your mesh is defined, into animated (with respect to C's joint angle) root space would be $(A_RB_RC_R)^{-1}A(t)B(t)C(t)$, where $M(t)$ is the local joint matrix at animation time $t$ and $M_R$ is the local joint matrix corresponding to the rest pose, the pose in which the mesh is defined, and thus $(A_RB_RC_R)^{-1}$ is C's inverse bind pose.


How can I draw this style of grid? I'm trying to write code (in any language) to render this 3D grid, matching the look and colors as closely as possible:

You can do it by only rendering a flat plane and then making a grid in fragment shader. This is actually a nice exercise.

To get these two color values, you can use color selector, which is available in any graphic software for example MS Paint

Some code hints:

vec2 coord = worldPos.xz / stepSize; //square size in world space
vec2 frac = fract(coord); //fractional parts of squares
//interpolation, grad is smoothness of line gradients
vec2 mult = smoothstep(0.0, grad, frac) - smoothstep(1.0-grad, 1.0, frac);
vec3 col = mix(col1, col2, mult.x * mult.y);


How to produce simple 2D illustrations to accompany geometry answers? In my answer about Affine Transformations I made some little illustrations to help the explanation.

How do you make an upload-able PNG quickly which expresses a geometrical or mathematical idea?

This question was contrived to support my postscript answer but is intended to solicit other solutions that may benefit the SE network more generally. But for credibility, an answer must link to a post where this technique has in fact been used, as well as a sufficient "howto" portion if it involves more than something obvious like "Select export from menu".

There are two meta questions here and here concerning how to scope this question in light of the precedences it may set for our site.

I use a variety of tools depending on the particular needs of the moment. I have a bias toward free tools, as I don't have access to things like Illustrator or Mathematica.

For general vector art purposes I use the free Inkscape. It operates on .svg files and can also render them out as high-quality .pngs or other bitmap formats. For example, here's a diagram I made in Inkscape to explain UV mapping:

For 2D function graphs, I often use the online graphing calculator Desmos, which has the handy capability of letting you share graphs online so others can view and modify them. You can also create sliders that can be varied to play with curve parameters in real time. For example, here is a configurable sharpening function.

For more complex graphing tasks that involve a bit of programming, I use Maxima, a free computer algebra system that also has graphing features. I used Maxima to create the graphs in this answer about Gaussian blurs:

This was a bit too complex for Desmos, since it involved programmatically creating a piecewise approximation of a Gaussian.

For really odd graphing tasks that aren't handled well by anything else, I've used Python with the svgwrite package to programmatically generate diagrams into .svg files. These can then be touched up in Inkscape if desired and rendered to bitmaps using Inkscape or ImageMagick. I used that workflow to create the images in this blog post on depth precision, like this:

In this case I wanted to not only graph a function, but place tick marks programmatically and nonuniformly along both the axes and the graph, to illustrate discrete depth buffer values.

And finally, if nothing else suffices, I'll use Python with numpy and pillow to render images pixel-by-pixel. That's how I created the Perlin noise images in this answer (with the help of the noise package too, in this case).


Is there ever a time when I should *not* use the GL* types? If I am writing my own function to check shader compile errors, or do anything that interacts with the OpenGL API but ultimately returns as a result of my application code, should I use the C++ void or GLvoid?

I.E.

GLvoid Shader::check_compile_errors(GLuint ComputeShader, std::string type)
{
    const GLint log_length{ 1024 };
    GLchar info_log[log_length];
    GLint success;

    if (type == "PROGRAM")
    {
        glGetProgramiv(ComputeShader, GL_LINK_STATUS, &success);
        if (!success)
        {
            glGetProgramInfoLog(ComputeShader, log_length, nullptr, info_log);
            std::cout << "ERROR::PROGRAM_LINKING_ERROR of type: " << type << "\n" << info_log << "\n ---- " << std::endl;
        }
    }
 // Here the return is implicitly void, but from C++ not OpenGL!
}

I realize here that GLvoid is typedef'd to void but I am wondering if there is any case where using the GL types could cause issues when it it is not explictly clear that the values are coming from OpenGL.

Another example would be writing an image to a file (from glReadPixels). If I am setting fields in the header which normally takes things in the form unsigned char should I change that to a GLuchar?

Yes, there's a time not to use GL types when interacting with OpenGL: GLvoid. There is absolutely no reason to use this type.

With everything else (assuming you're using a "standard" OpenGL loader), you should use OpenGL's types when talking to OpenGL. That is, the values you pass to OpenGL functions should be in OpenGL's types. This may mean that a function that talks to OpenGL may need an OpenGL data type in its function parameter list. This should be avoided where possible, to prevent you from creating dependencies between OpenGL code and non-OpenGL code.

However, if you don't mind having large portions of your application have dependencies on OpenGL, you may use OpenGL's data types throughout your code.

OpenGL's data types are specified to have a particular bitdepth. As such, they are (conceptually, if not strictly C++-legally) pointer-interconvertible to certain types defined in <stdint.h>/<cstdint>. Indeed, many OpenGL headers will check for the existence of this header and use it wherever it is available to define OpenGL's data types. As such, if you want a fixed bitdepth integer, you have a choice besides using OpenGL's data types.

  I am wondering if there is any case where using the GL types could cause issues when it it is not explictly clear that the values are coming from OpenGL.

OpenGL is C, not magic. A function in C cannot have any control over the return value of the function that calls it. Even a C++ function cannot; at best, it can throw an exception, but this still doesn't have anything to do with its caller's return value.


Are the benefits of a Metal early depth test limited to one draw call I should preface with currently this question is mostly asking about Metal and iOS GPUs. That being said if this is GPU or OS specific and you can go into detail about other platforms that would be great to as this application will eventually be cross platform.

I am drawing some geometry in two draw calls.

Opaque geometry is drawn. Discards happen. No particular triangle draw order. Fragments in this draw call occupy only 5% of the screens pixels.
Transparent geometry is drawn. No discards happen. Triangles are drawn in depth order. Fragments in this call occupy a bulk of the screen pixels.

The question is essentially should I convert draw call 1 to ordered triangles with alpha=0 where I would generally discard. There will be a cost to ordering the triangles in draw 1 however it will not be two costly.

The reason I am unsure is that I know the second you call discard on a fragment you remove the possibility of Metal preforming early depth test which is allegedly quite an optimization. What I dont know is if discards disable early depth test on draw call 1 do they also disable the optimization for draw call 2's geometry?

Does the fragment stage happen after each draw calls vertex phase or after all of them?

Do you think I should explore sorting the triangles and using alpha blending on draw call 1? Again this would be a negligible CPU cost would it actually help the GPU?

I don't know what Apple have done with Metal per se, but as the iOS devices have PowerVR-derived GPUs, normally I would think that doing as much opaque-only drawing "up front" would usually be beneficial, but if, as in your case, only 5% of the screen is covered by opaque geometry, I'm not sure how much the software overhead will affect things.

  The reason I am unsure is that I know the second you call discard on a fragment you remove the possibility of Metal preforming early depth test which is allegedly quite an optimization. What I dont know is if discards disable early depth test on draw call 1 do they also disable the optimization for draw call 2's geometry?

With PowerVR devices, a depth test can "always" be done as the first step in the pipeline, so any geometry that was submitted earlier in the scene can cull later things prior to fragment shading.

With alpha-tested (AKA "punch-though"), the Z-test can thus be done against previous geometry, but there may be a cost of then doing the texturing/shading and feeding that back to update the depths for subsequent geometry.

If you aren't doing alpha-test/discards, and are doing your own sorting, then this feedback is not required (but obviously there may be more shading/texturing work to perform).

(Such a pity, though, that we removed the OIT feature from the early PowerVR hardware)

  Does the fragment stage happen after each draw calls vertex phase or after all of them?

I'm not sure what you are asking, but for a given scene, all the vertex shading (across multiple calls) is performed**, the geometry buffered, and only then rendering commences on the entire scene.

**There are some caveats here but that probably doesn't affect you.

  Do you think I should explore sorting the triangles and using alpha blending on draw call 1? Again this would be a negligible CPU cost would it actually help the GPU?

Are you asking "should I profile/time the results and see what is best for my use case?".  Probably "yes".


Do GPU performance improve when rendering at native or half resolution, compared to fixed scaling? This might be an odd question, but I was wondering if you could improve GPU performance with games, comparatively, when using doubles/halves of the native resolution, instead of an odd ratio of that.

Lets compare the situation with a metaphor first. If I'd give you a standard A4 sheet of paper, I bet you can more easily fold it in half than 1/3rd. Same goes when I'd give you two sheets of A4 and ask you to create a A3-size rectangle, instead of a not-A3-but-larger-than-A4-shape.

To go back to the actual situation: My Macbook has a native resolution of 2880x1800. But the game offers a variety of resolutions, such as 1440x900, 1680x1050, 2048x1280, 2560x1600 and for some reason also 3360x2100, which is above the native resolution..

If we'd include the sheet of paper thought in resolution scaling with games, compared to the native resolution, I'd say 1440x900 could run even better than 1280x800, because the GPU could simply say: reduce all pixels by 50%. Not only is that an easier calculation, but it's also much easier to interpolate, if that's the correct word for it to calculate how the pixel should look like.

Sudden thought: This does assume the game environment is rendered at the native resolution and thereafter downscaled to the set resolution, which is probably wrong. That said, I do think it'd be easier to render at half the resolution than something like 40%.

This idea does apply to Photoshop with increasing and decreasing the resolution: Always increase with halves and doubles. So why not games?

I'm not only very interested in the actual answer, but perhaps makes it easier to set a resolution for a battery vs performance question :)

GPUs can efficiently scale an image by an arbitrary amount (within limits - display options fall within those limits by design) either using a 3D rendering operation or as the signal is sent from GPU to the display. Both of these paths have fully dedicated hardware for the arbitrary resizing and are not likely to be optimized for doubling or halving. Both support a technique called "bilinear filtering" though advances in display hardware may provide higher quality.

If a game is run at non native resolution on a laptop (TVs and desktop displays have their own scalers, but laptop displays generally rely on the GPU), one of these two methods will be used. Display scaling is essentially "free", so it is preferred and the lower the resolution the game is originally rendered the faster the frame rate (or the lower power consumption) - with no special performance benefit for, say, doubling. If a 3D operation is required (say, due to a limitation or issue in the display scaler), the scale operation is not free, but is going to go up with the source and destination sizes smoothly, with no special casing at doubling/halving.

The game may look far better at certain scaling factors, but that's up to the user to decide.

Edit: The intuition behind the question is correct: if GPUs didn't have dedicated scaling hardware, doubling/halving could implemented more efficiently than arbitrary scaling, either in shader code or via a slightly simpler hardware design - though the 3D rendering operation is a requirement of graphics APIs and display scaling is a requirement from systems vendors.


Is it possible to set visible and color for all 12 edges(spines) of the figure (cube) in matplotlib? In 2d, matplotlib plots all 4 edges(spines) of a figure.

In 3d, matplotlib plots 9 out of 12 edges of a cube.

Is it possible to set visible and color for all 12 edges(spines) of a cube in 3d?

I tried ax.xaxis._axinfo['juggled'] = (2,0,1) and found that is something live move instead of setting visible or color.

I also tried ax.collections, didn't work either.

Any clue?

matplotlib's 3d rendering settings are not yet complete (it's meant to be a 2d renderer program). I don't think at this stage it has this option without big hacks to the code locally (i.e. done by yourself). Mathematica offers this setting naturally.


Is there any way to generate primitives in a geometry shader without any input geometry? A few years ago I tried to implement this GPU Gem in OpenGL to generate 3D procedural terrain using Marching Cubes. The article suggests to implement Marching Cubes in a geometry shader to maximum efficiency. This means I need to run the shader once for every voxel in the domain, and it'll generate the entire geometry in that cell.

One problem I stumbled upon was how to get the geometry shader running without actually having anything to render outside of that shader. My solution (which seemed rather hacky) was to render a point in each cell, discard it with the geometry shader and emit my triangles instead. I never found a proper solution and this workaround remained in the final code.

So, is there anyway to tell OpenGL to start a rendering pass from the geometry shader without any input geometry? Or will I always have to send some dummy points to the GPU to get things running.

No, there is not really a way to do that.

A geometry shader invocation requires an input primitive and generates 0 or more output primitives. Without an input primitive, there is not really a way to actually invoke the geometry shader. Of course you can stretch the limits of the geometry shader's maximum number of output primitives for each input primitive (don't know the practical limits right now, should be on the order of thousands maybe). So you maybe could generate 1024 triangles for each point, but you always have to have some input primitives.

What you don't need however is an actual notion of geometry. You don't really have to render 3D points at any reasonable location, they could as well have just some abstract index or texture coordinates or whatever as attributes and not necessarily a meaningful 3D position. Noone dictates what attributes your vertices have. And you can even render vertices without any attributes. But you have to render some primitives to invoke the geometry shader on them, even if those primitives don't have any actual attributes (how you then compute your output geometry in the geometry shader is a different question, though).

But what you actually did, rendering a point for each grid cell and generating the marching cubes triangles for that cell from it, is exactly the straight-forward approach. Of course what attributes this cell contains is up to you, it could be a 3D position, a texcoord into a 3D texture, whatever, but those are the grid cells you render. Purely semantically spoken, you don't actually "discard" those points and then "replace" them by triangles, you "convert" each point into a set of triangles. That's exactly what the geometry shader is for and there's nothing "hacky" or "improper" about it. Noone says the geometry shader has to generate the same output primitive type as was input to be "proper".

What you can do to achieve a largely input-less way of rendering your voxel grid (and this might be what you were actually asking for) would be to just draw a set of attributeless points. This means you don't need any attribute arrays at all and just disable them all and invoke a simple glDrawArrays with the number of cells you need. Then in the vertex shader or the geometry shader you can generate the necessary 3D grid cell index with a little index magic from the input vertex ID (i.e. gl_VertexID, which is the only information you have) and then compute your marching cubes geometry from a lookup into the 3D volume texture (or whatever datastructure).

So, in retrospect I should relativate my statement from the beginning: You can't generate primitives without any input primitives, but you can generate them without any input geometry.


Shader or formula that distorts inward I am looking for a shader (or at least the right terminology) for crushing an image. It should be something very similar to barrel distortion, but instead of the center crushing outward to the edges, I want to edges to crush inward to the center.

So, this image:

instead of being deformed like this:

Should rather be deformed like this:

This will be used for a Godot project, but even regular shading language would be tremendously helpful. I tried modifying this gist to no avail.

Edit

Here is the final result using the accepted answer's solution:

Your example images do not make the desired effect very apparent, but the opposite of a barrel distortion is a pincushion distortion:

I found a fisheye shader on ShaderToy which produces the following image:

By slightly modifying the shader code:

// This shader is based on cafe's fisheye shader:
// https://www.shadertoy.com/view/ll2GWV

vec3 checker(vec2 uv){
    return vec3(abs(floor(mod(uv.x*10.,2.))-floor(mod(uv.y*10.,2.))));
}

void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
    vec2 uv = fragCoord.xy*2. / iResolution.xy-vec2(1.);

    //------------------------------------------------
    // To use in Godot, port this section:
    //------------------------------------------------

    // I picked these somewhat arbitrarily
    const float BARREL = -1.0;
    const float PINCUSHION = 0.1;

    float effect = PINCUSHION; // Set effect to either BARREL or PINCUSHION
    float effect_scale = 1.0;  // Play with this to slightly vary the results

    /// Fisheye Distortion ///
    float d=length(uv);
    float z = sqrt(1.0 + d * d * effect);
    float r = atan(d, z) / 3.14159;
    r *= effect_scale;
    float phi = atan(uv.y, uv.x);

    uv = vec2(r*cos(phi)+.5,r*sin(phi)+.5);

    //------------------------------------------------
    // end relevant logic
    //------------------------------------------------

    fragColor = vec4(checker(uv),1.);
}

We can easily obtain the opposite effect:

If this is not the exact effect you're looking for, I recommend playing around with the code a bit yourself. Change some signs from + to -, add scale factors and explore the possibilities. You may even stumble across a more interesting effect than what were you looking for.


Buffer of arrays in HLSL compute shader I'm trying to follow along with a tutorial that uses an OpenGL compute shader, but I'm using D3D11. What they do is:

layout (std430, binding = 0) buffer indices {
    int j[];
};

I have no experience with this and very little with directX, but that looks like a buffer of integer arrays. I've looked around and read that the equivalent to this OpenGL buffer in D3D11 is an RWBuffer. Using a compute shader, this is what I've tried. Note, I didn't redefine them in my code as they appear here, I just listed them together here.

RWBuffer indices
{
    int j[];
}

RWBuffer<{int j[];}> indices;

struct index
{
    //requires explicit length, which is why this doesn't work
    int j[];
}
RWStructuredBuffer<index> indices;

But, none of those worked. The end goal is to access them like so:

//id is SV_DispatchThreadID
indices.j[int(id.y)]

This seems like an inappropriately simple question to post, but I wasn't able to find documentation, even on MSDN, on defining buffers and what's allowed and not, etc.

It looks to me like you want a buffer containing a single array of integers (not a buffer containing multiple arrays, whatever that would mean). So, you should be able to just do this:

RWBuffer<int> indices;

then access it like this:

indices[int(id.y)]


Algorithm to find the center of a Bezier curve I need to find the center of a Bezier curve to rotate it. I have a list of all points (control points, beginning and end, all points on the curve itself). How would I go about finding it's center?

Bézier curves are mathematical entities and have no clearly defined center. One can in fact define many different things as the center of the Bézier curve. I have tried to depict some of the possible centers in image 1. More than this do exist.

Image 1: Some of the possible centers of a single span Bézier curve

In practice nearly all graphics applications geared for drawing use the center of the local bounding box (BB) as their center. Animation software usually have an additional concept of pivot so they use the ask user approach, if no input is made they often revert to BB center or simply local coordinate center. This is probably because the BB needs to be calculated anyway and getting its center is pretty easy to do (see A Primer on Bézier Curves).

The center of gravity metrics are also somewhat natural especially in an animation context, though nastier to compute. Easiest being to discretize the data and do the computation on the discrete input. This said some closed form solutions are possible for the curve center of gravity, but it's not a very nice equation to formulate and simplify.

Then we have the on the curve points: the midpoint by arc length and the point where the $t$ parameter is 0.5. In my mind the $t$ param is often problematic though easy to compute, and it loses meaning when you chain multiple Béziers after each other for a polybézier. The center of length of course is only natural as long as the curve is not closed.

We also define other possible centers, the center could be at the curve center of gravity of the hull, the average of the control points or the control cage BB center. Although in practice these do not seem to work out very well.

Please note: Although the curve in picture 1 shows the BB center quite near to some natural centers this this is not always the case for more complex curves and especially polybéziers.


What can be the cause of some strange ray shadows artefacts on a perfectly flat surface? First of all I have to say that I am really new in the world of Computer Graphics. I apologise if some of my questions may appear trivial.
I am currently using a non-commercial ray tracing and physically based rendering software. I didn't have any problems with this until now : I have a scene where an .obj file representing a satellite is placed surrounded by a environment map light.
Here is an exemple of my problem appearing on the solar pannels :

I have no idea where these strange forms could come from. I have to say that for this picture, the .obj file representating the solar pannel is a very simple parallelepiped. I also tried to suppress the other face of the object but nothing changed.
I saw this exchange (Cause of shadow acne) that could maybe be related to my problem, but if someone has an idea of what could be the cause, I would be happy to hear it!
Thank you to all and have a nice day.

It appeared that it was exactly the problem explained in the link of my original post (Cause of shadow acne).
The shadow function offset has been increased and the problem is solved!


How to implement Texture Baking? So currently I want to implement radiosity in my engine. After figuring out the amount of light that bounces between elements/disks/patches (I've been told disks are the best - correct me if I'm wrong) how do you bake it into a texture?

Do I set the render target as the size of my texture and modify my  shader to work with UV coordinates? This is really confusing me so is there another way? Do I even need to bake the light into the textures, is hardware good enough these days?

Yes, you should set the lightmap texture as the render target and use UV-coordinates as vertex position coordinates in your shader to bake lighting into lightmaps. This way you'll write the baked lighting data of triangles into the correct lightmap texels that are used for rendering the triangles on the screen. The UV-map should be unique and no triangles should overlap in the UV-space. The lightmap data needs to be also expanded so that when you later use bilinear filtering to fetch the data from the texture that you don't end up having dark borders following triangle edges at UV seams.

Instead of baking indirect illumination into surface lightmaps, you could also use volume textures. Nice thing is that the with volume textures you'll have the indirect illumination data available everywhere in the scene and you don't need unique UV-mapping, but you'll also have less resolution due to the added memory usage.

If you need to bake indirect illumination into textures depends on the complexity of the scene, quality of the lighting and performance/mem usage you want to achieve, so there's no easy answer.


Confused about material layering I am a little bit confused about the various flavors of material layering in both online and offline rendering.  I had naively assumed (after using the Mix Shader in Cycles and a few minutes in Substance Painter) that material layering was just a simple mask channel that blended between the final colors, sort of like texture splatting for terrain.

But reading this question I see that the actual shader inputs are interpolated, rather than the final colors:

Material Layering

Is this the same process for offline renderers?  Reading the man page for RenderMan it seems like it is, as they specify only similar Bxdfs can be mixed:

https://rmanwiki.pixar.com/display/REN/PxrLayerSurface

Are there any papers that describe the basics?  I found this from SIGGRAPH last month, but it confused the hell out of me:

http://blog.selfshadow.com/publications/s2017-shading-course/drobot/s2017_pbs_multilayered.pdf

Lastly, are there any contexts in which material layering is simulation of multiple physical interfaces?  Or is that referred to something else, like nested dielectrics?

There is no single method of layering materials. Some implementations, as you have observed, simply interpolate shader input parameters. This helps especially in contexts where code divergence comes with performance penalties (GPUs for example), since all material layers use the same code path and only have different data. Other blend between the output of the individual shader layers.

More complex approaches take the interface between material layers into account, as well as layer thickness. A recent publication describing such a method is "A Comprehensive Framework for Rendering Layered Materials" by Jakob et al.


Why do some videos look smoother and others don't on the same monitor I have a Dell monitor with 60 Hz refresh rate and therefore games playing on it are not very smooth as compared to 120 hz.
But when I open videos on YouTube of the same game on the same monitor, some of those videos run extremely smoothly as if it's a 120hz display.
Why is that so ?

As you have now mentionned that your computer can actually keep up (at 150 fps no less), I suspect you have a case of temporal aliasing.
The problem is that 150 is not a multiple of 60. Let's say we look at one tenth of a second. That's 15 frames generated by your computer but only 6 can be shown on your screen. The frames you will see will probably be something like:
#1 - #3 - #6 - #8 - #11 - #13
Notice how the increment is +2, +3, +2, +3. This will cause animation to not look smooth. There is no steady increment which will let you display 60 frames out of 150.
As I wrote in the comments below, forcing the game to run at "only" 120Hz should help as the screen should show every other frame.
Enabling vsync is even more reliable but it seems your game does not deal well with latency then.
By the way, that's why serious gaming monitors now have adaptive refresh rates. They just show the frames when they're ready and have none of the above issues.


Why does the 1/r² term appear with point sources? I'm currently reading some books on radiometry. They mention that radiance is constant along a ray. It doesn't change with distance.
However, I've seen some raytracer and they put the 1/r² factor when they deal with point sources. I don't get why. I didn't find a great explanation on the Internet.

The concept of a point source is an approximation. Physically, light sources are extended objects and emit light from every point on their surface; but when you're far enough away (i.e. the distance to the source is large compared to its size) it's useful to approximate it as a point source.

You can get the $1/r^2$ law out of it as follows. Imagine a spherical area light with some radius $r_\text{light}$, and you're looking at it from a distance $r$ away. Then, we can approximate the solid angle that it subtends from your point of view as the area of a circle of radius $r_\text{light}/r$ (just using similar triangles). This area will be $\pi (r_\text{light}/r)^2$, so it's proportional to $1/r^2$.

Note that this approximation becomes exact in the limit $r_\text{light}/r \to 0$, i.e. when the light source is very far away or very small. It breaks down if the source is too large or too close.

If the source emits a constant radiance from every point on its surface, then when you integrate over solid angle in the rendering equation, you'll get a total irradiance proportional to $1/r^2$. In order to approximate it as a point source in a renderer, we skip the integration and just add an irradiance proportional to $1/r^2$ directly.


Operator associativity and 4x4 matrices - performance question During vertex processing with 4x4 matrices, we might stack multiple transformations like projection, model-world, world-camera, etc. by doing something like this:

$$v_{final} = T_N \cdot ... \cdot T_1 \cdot T_0 \cdot v$$

Now from the GLSL specs (section 5.1), I get that the operator associativity is "left to right". So this:

$$v_{final} = T_2 \cdot T_1 \cdot T_0 \cdot v$$

is equivalent to:

$$v_{final} = ((T_2 \cdot T_1) \cdot T_0) \cdot v$$

However, as matrix multiplication is associative, the result does not change if we write:

$$v_{final} = T_2 \cdot (T_1 \cdot (T_0 \cdot v))$$

While both versions yield the same result, the second version should (at least in theory) be much cheaper to compute since it incorporates 3 matrix-vector multiplications while the first version does 2 matrix-matrix multiplications and a final matrix-vector multiplication.

So my question is, does enforcing the order by adding extra parenthesis really boost performance or is the GLSL compiler usually smart enough to figure out, that the order of operations does not matter for the result and he can reorder the executions for performance even though the specs say "left to right" associativity?

Additional Note:
I specially mentioned 4x4 matrices, because if you use 3x3 matrices or quaternions, you have to use parenthesis anyways because of the addition of the translations.

As Olivier mentioned, it's not safe in general to apply algebraic optimizations to floating-point math, as floating point doesn't actually obey associativity rules due to roundoff error. $(a + b) - b$ will not be equal to $a$, if the intermediate result $(a + b)$ had to be rounded; nor will $a \cdot (b + c)$ be equal to $a \cdot b + a \cdot c$ in general, for the same reason.

That being said, shader compilers are typically very relaxed about floating-point math optimizations. They behave like using -ffast-math on a C++ compiler, where they make various simplifying assumptions—including that algebraic operations are associative—in order to optimize more aggressively. So it's not out of the question that a GLSL compiler could actually make the optimization you describe.

That said, GLSL compilers vary wildly in quality; there is no canonical GLSL compiler, every GPU vendor implements their own (for OpenGL at least—in the case of Vulkan there are a couple standardized options), and who knows what level of optimizations they might or might not apply under various circumstances. If you wish the code to be optimized in a particular way, better to write it explicitly that way.

One other point,

  the second version should (at least in theory) be much cheaper to compute

It's only cheaper if you're transforming just a single vertex. If you want to transform thousands of vertices by the same matrix, it will be cheaper to multiply all the matrices together up front; then each vertex only needs to be transformed by a single 4×4 matrix instead of several.


ShadowMap peter-panning and wrong direction I am creating Forward-Rendering DX11 engine and I have some troubles with shadow-mapping algorithm as it doesn't work as supposed.

As you can see - shadow is peter-panning and looks totally not releated. Basically as I have tested, comparision in PixelShader has no effect. Only projected shadowmap texture is returned, if statement doesn't do anything.

MyVertexShader:

PixelInputType ColorVertexShader(VertexInputType input)
{
    PixelInputType output;
    float4 worldPosition;

    input.position.w = 1.0f;
    worldPosition = mul(input.position, worldMatrix);
    output.normal = input.normal;

    // Calculate the position of the vertex against the world, view, and projection matrices.
    output.position = mul(input.position, worldMatrix);
    output.position = mul(output.position, viewMatrix);
    output.position = mul(output.position, projectionMatrix);

    //Calculate object seen by light
    output.lightViewPosition = mul(input.position, worldMatrix);
    output.lightViewPosition = mul(output.lightViewPosition, g_lightViewMatrix);
    output.lightViewPosition = mul(output.lightViewPosition, g_lightProjectionMatrix);

    //Calculate light-object vector
    output.lightPos = g_lightPosition.xyz - worldPosition.xyz;
    output.lightPos = normalize(output.lightPos);

    return output;
}

MyPixelShader:

float4 ColorPixelShader(PixelInputType input) : SV_TARGET
{
    float4 color = float4(0.0, 0.0, 0.0, 1.0);
    float depthValue;
    float lightDepthValue;
    float bias = 0.001f;
    float4 textureColor = float4(1.0, 1.0, 1.0, 1.0);
    float4 ambientColor = float4(1.0, 0.0, 1.0, 1.0);
    float2 tex;
    float lightIntensity = 0.0f;

    input.lightViewPosition.xyz /= input.lightViewPosition.w;

    tex.x = input.lightViewPosition.x / 2.0f + 0.5f;
    tex.y = -input.lightViewPosition.y / 2.0f + 0.5f;

    return depthValue = shadowMap.Sample(SampleType, tex).r;
    color = float4(depthValue, depthValue, depthValue, 1.0f);

    lightDepthValue = input.lightViewPosition.z;// / input.lightViewPosition.w;
    lightDepthValue = lightDepthValue - bias;
    if(lightDepthValue < depthValue)
    {
        lightIntensity = saturate(dot(input.normal, input.lightPos));

        if(lightIntensity > 0.0f)
        {
            // Determine the final diffuse color based on the diffuse color and the amount of light intensity.
            color += (ambientColor * lightIntensity);

            // Saturate the final light color.
            color = saturate(color);
        }
    }
    return color;
}

I found out that simply removing minus sign in shader in converting coords to [0, 1] solved direction problem. But scale is still off - here is an example:

void LightClass::GenerateViewMatrix()
{
    XMVECTOR eyePos = { m_position.x, m_position.y, m_position.z };
    XMVECTOR focusPos = { m_lookAt.x, m_lookAt.y, m_lookAt.z };
    XMVECTOR upVec = { 0, 1, 0 };

    m_viewMatrix = XMMatrixLookAtLH(eyePos, focusPos, upVec);
}

void LightClass::GenerateProjectionMatrix(float screenDepth, float screenNear)
{
    float fieldOfView = (float)3.14f / 2.0f;
    float screenAspect = 16.0f / 9.0f;

    m_projectionMatrix = XMMatrixPerspectiveFovLH(fieldOfView, screenAspect, screenNear, screenDepth);
}

Edit [Actual solution - ignore previous posts]:

I followed PaulHK solution (thanks, it's second time you've helped me) about checking once again View/Projection matrix. Basically my calculations were wrong. I debugged my convertions from model to lightspace and changed Vertex Shader from:

output.lightViewPosition = mul(input.position, worldMatrix);
output.lightViewPosition = mul(output.lightViewPosition, g_lightViewMatrix);
output.lightViewPosition = mul(output.lightViewPosition, g_lightProjectionMatrix);

To correct:

matrix lightViewProj = mul(g_lightProjectionMatrix, g_lightViewMatrix);
output.lightViewPosition = mul(input.position, lightViewProj);

That simple changed fixed my results. With simple Pixel shader code below I am getting this results:

input.lightViewPosition.xyz /= input.lightViewPosition.w;

tex.x = input.lightViewPosition.x / 2.0f + 0.5f;
tex.y = -input.lightViewPosition.y / 2.0f + 0.5f;

depthValue = shadowMap.Sample(SampleType, tex).r;
return color = float4(depthValue, depthValue, depthValue, 1.0f);

However, even if it seems correct, my suspicion is that I need to compare my shadowmap to light Z-buffer so I modified my Pixel Shader and added little tint:

input.lightViewPosition.xyz /= input.lightViewPosition.w;

tex.x = input.lightViewPosition.x / 2.0f + 0.5f;
tex.y = -input.lightViewPosition.y / 2.0f + 0.5f;

depthValue = shadowMap.Sample(SampleType, tex).r;
color = float4(depthValue, depthValue, depthValue, 1.0f);

lightDepthValue = input.lightViewPosition.z - bias;
if(lightDepthValue < depthValue)
{
    lightIntensity = saturate(dot(input.normal, input.lightPos));

    if(lightIntensity > 0.0f)
    {
        // Determine the final diffuse color based on the diffuse color and the amount of light intensity.
        color += (ambientColor * lightIntensity);

        // Saturate the final light color.
        color = saturate(color);
    }
}
return color;

I may not be right, but without this checking, you would get errors with more complicated shadows. Because my previous PixelShader code basically just project shadowmap on plane without any actual depth checking. Therefore it would lay shadowmap on everything it can without even checking if that's correct.


What is the difference in overlay and framebuffer? I have been trying to understand display and graphics. I feel that it's hard to make a logical image of the terms. I understand this:

Frame buffer is a dedicated memory that has the image/video/3dimage or video to be shown, in pixel format.
Overlay is some kind of hardware that is used with the frame buffer.
Surfaceflinger is a composition manager. It takes rendered data from the GPU, and composites them in one surface. That means, it uses the underlying hardware and driver to do all this composition.
Composition means blending, mixing of all the surfaces.
Overlay, plane, framebuffer and sprite? What does all this do, or how to combine them and understand?

Any book, links will be appreciated. A lot of books go to graphics way, instead of taking one particular SoC and explain display first then graphics in detail.

As you've understood, the framebuffer is an array in memory that holds all the pixels to display on the screen. On a desktop PC, it's probably special memory on the graphics card, but in a SoC with one memory shared by GPU and CPU, it's probably a normal memory allocation that the display controller uses DMA to read from.

The display controller is a piece of hardware that reads pixels from the frame buffer and displays them on the physical screen.

Many display controllers support the use of an overlay, which is like a very simple window. Sometimes they're fixed-size, but sometimes they're masked off. In this case, then there might be a secondary framebuffer that covers the whole screen. When the display controller reads a pixel of a particular colour from the primary framebuffer, then instead of displaying that pixel, it substitutes the corresponding pixel from the secondary framebuffer. It's like the chroma key or green screen technique used in movie effects and weather forecasts.

That's one particular configuration: sometimes the secondary framebuffer is much smaller than the primary framebuffer, and a particular register in the display controller controls the dimensions and offset.

The overlay technique is commonly used for displaying video streams. The frames of video might be coming from camera hardware or from video decoder hardware. Using an overlay means that this hardware and the GPU that's producing the rest of the screen (such as camera or playback GUI) aren't trying to access the same memory at once. The overlay might also be in a different pixel format to the primary framebuffer.

The overlay technique only lets you do this simple substitution, though (and maybe some scaling), not anything complicated. You can't make the frame of video spin around and shrink to an icon when you're transitioning between different applications. You can't alpha-blend the video with the primary framebuffer. Also, the number of overlays is limited by the hardware: usually one, but some controllers support two.

Composition in this context refers to something the window manager does. It takes the frames from each application on the screen (the system GUI is also an application), and produces the final frame. On a smartphone, this usually means just taking the current foreground app and the system GUI (notification bar and navigation bar) and drawing them next to each other. But when there are transitions between apps, or if you use the "recent tasks" button, the situation is more complicated. The compositor also has to handle choreographing applications so the screen gets updated at 60 Hz even if applications are drawing at different rates.

Surface Flinger is Android's compositor. It uses OpenGL ES to do the things described in the previous paragraph. It's also in charge of creating a framebuffer for each application to render into. (These framebuffers show up as textures inside Surface Flinger, using EGLImage.)

Android also detects when there are simple cases that the display controller on the phone can handle as an overlay, so that it can switch to the faster, more energy-efficient path of using the overlay instead of using Surface Flinger's full GLES-based composition.


Rendering equation for spherical lights I am currently implementing spherical lights in my DirectX game engine. I decided to build the shading formula from the classical rendering equation. Here is what I get:

The simplifications show that the cosine term should be removed for spherical lights since power is a Flux.
Here the problem. While looking for recourses online I found that engines always multiply by the cosine term, even for spherical lights.

Is it wrong or am I missing something?

If you just want to explicitly sample an area light, then here's the general procedure you should follow. Pick light $i$ out of $L$ lights with some probability $p_i$ (the other probabilities being $p_1,...,p_L$, a light may be picked through inverse transform sampling). Pick a point $\pmb{y}$ on the surface of the light with some probability $q_i(\pmb{y})$. Then your estimator is:

$$I_N = \frac{1}{N}\sum_{k=1}^{N}\frac{1}{p_{i_k}q_{i_k}(\pmb{y}_k)}f(\pmb{z} \leftarrow \pmb{x} \leftarrow \pmb{y}_k)L_e(\pmb{y}_k\rightarrow\pmb{x})\frac{\cos\theta_x\cos\theta_{y_k}}{\|\pmb{x}-\pmb{y}_k\|^2_2}V(\pmb{x},\pmb{y}_k)$$

Note that this is only for direct illumination at point $\pmb{x}$. I have generalized it to a secondary estimator using $N$ samples, so essentially you pick $N$ points $\pmb{y}_1,...,\pmb{y}_N$ on possibly different lights. This is an estimator that can directly be derived from the area formulation of the rendering equation:

$$L(\pmb{z} \leftarrow \pmb{x}) = L_e(\pmb{x} \leftarrow \pmb{z}) + \int_{\Omega}f(\pmb{z} \leftarrow \pmb{x} \leftarrow \pmb{y})L(\pmb{y}\rightarrow\pmb{x})\frac{\cos\theta_x\cos\theta_y}{\|\pmb{x}-\pmb{y}_k\|^2_2}V(\pmb{x},\pmb{y})\,dA(\pmb{y})$$

The area formulation can be derived from the solid angle one, by using the identity:

$$d\omega = \frac{\cos\theta_y}{r^2}dA$$,

as well as:

$$L_i(\pmb{x}, \omega) = L(r(\pmb{x},\omega) \rightarrow \pmb{x}) = L(\pmb{y} \rightarrow \pmb{x})V(\pmb{x},\pmb{y})$$


GLUT OpenGL - Drawing Shapes using only GL_POINTS I'm working on an assignment and I need to draw using only GL_POINTS. I realize this is an expensive approach but it's for the homework, so no GL_TRIANGLES / GL_POLYGON / GL_LINES etc.

First, I'm trying to understand the concept of using only points. Say I want to draw a square that's 100 x 100 pixels. Would I need four for-loops drawing 100 pixels each in straight lines to create the square? What if I want to fill the square with color?

I understand drawing a square using GL_POLYGON as that's fairly straight forward. We're using the GLUT library just to draw shapes with points.

I reckon your instructor is encouraging you to implement low-level rendering algorithms.

The obstacle to draw shapes using just points is the contradiction between continuous and discrete. Shapes are logically continuous if in its area every pixel is shaded. But you can not draw an actually continuous line using discrete points because infinite points are needed. If you draw points directly in the "world", you can not guarantee every pixel on the screen can be covered after projection, unless you set the interval small enough(which may be of quite low performance).

My advice is to maintain the world coordinates of the vertices of the shape but not draw it. Get the "projected" coordinates on screen and fill the area later using DDA or some other interpolation algorithms. "Projected" is the coordinates obtained by the three matrices(Projection, View, Model), I think you've heard of these. So in the filling process GLUT is used to draw points on the 2D screen, and cover every pixel in the area.


How do you interpolate UV mapping coords in a quad tessellation evaluation shader? Does anyone know who to interpolate UVs in a quad tessellation evaluation shader? I can interpolate the position easily like so :

vec4 p1 = mix(gl_in[0].gl_Position,
        gl_in[1].gl_Position,
        gl_TessCoord.x);
vec4 p2 = mix(gl_in[2].gl_Position,
        gl_in[3].gl_Position,
        gl_TessCoord.x);
gl_Position = mix(p2, p1, gl_TessCoord.y);

But I can't figure out how to interpolate the incoming UV coordinates. I need to do so to pass them on to the fragment shader.

With perfect quad UVs from 0 to 1, I can use gl_TessCoord.xy. Unfortunately that doesn't work when the UVs aren't mapped 1:1 to a texture.

Thanks

The problem was else-where, the same interpolation works for UV coords.

vec2 uv1 = mix(teUv[0], teUv[1], gl_TessCoord.x);
vec2 uv2 = mix(teUv[2], teUv[3], gl_TessCoord.x);
fUv = mix(uv2, uv1, gl_TessCoord.y);


How to create 2D (directional) noise? I'm just getting started trying to understand noise generation algorithms. What I'm trying to achieve is to get a 2D (or 3D) grid of random directional vectors (again, 2D or 3D) according to a noise distribution like Perlin, ie. not just a grid of totally randomly generated vectors (which would be easy to do).

Before starting to figure this out, I had assumed that 1D noise functions would be those that produce a series of float values (ranging from -1 to 1, for example), and that 2D noise functions would produce float2 values, 3D noise would produce float3 values, and so on.

However, searching for so-called 2D noise functions, it seems that they are used to produce a 2D grid (ie. texture) of single float values. However, what I'm looking for is a 2D grid of 2D (float2 / vector2) values, or a 3D grid of 3D (float3 / vector3) values.

From my (still pretty limited) understanding of a gradient-based noise like Perlin, I feel like I could probably hack something together from an existing Perlin function - doing a bi-linear/tri-linear interpolation of the surrounding gradient vectors which are ordinarily used for the subsequent dot product in Perlin noise. But I have no idea if that's the correct approach.

Are there existing libraries/methods for something like this?

Why don't you just use Perlin noise twice on the same grid, or volume? Each with slightly different parameters (a phase shift, or different pseudo-random vectors). In this case both component of your float2 are smoothly defined by a Perlin Noise field. A float2 $v$ could be defined as $v = \{P(u,v), P(u+0.5, v+0.5)\},$ where $P(u,v)$ is the Perlin noise function. However, this will make the $x$ and $y$ component of $v$ very close depending on the scaling of the grid values.


Detect collision for non axis-aligned bounding boxes in my application, I needed to detect collision between 2 3D objects. To make it simple I can represent a object as a bounding box. So actually the task reduced to detect collision between 2 boxes in 3D space. The problem that I cannot make these bounding boxes to be axis aligned.
The objects look like the following image:

Unfortunately all the algorithms I've found in the Internet suppose that all the boxes are axis-aligned. It seems to me that this is a common mathematical problem and can be solved using some well known algorithms. Unfortunately I'm not a mathematician so I would be very grateful if someone can offer a solution to this problem.

Write two functions: (1) point in 3D box, and (2) segment intersect rectangle in 3D. Point in box is a conjunction of six left-of-plane checks. For (2), solve for the point of intersection between the line containing the segment and the plane containing the rectangle. Then check if that point is on the segment, and if so, in the rectangle.
Then box $B_1$ intersects box $B_2$ if any corner of $B_1$ is inside $B_2$, or if any of the $12$ edges of $B_1$ intersects any of the $6$ faces of $B_2$.
This can be sped up a bit by ordering the tests (for example, if no edge of $B_1$ intersects a face of $B_2$, then you need only check one corner of $B_1$ inside/outside $B_2$).


Polygons versus curve primitives in software rendering Most 3D video games since the nineties have used hardware rendering based on polygons. Why polygons? They work well for some things, but not so well for others, e.g. a human figure rendered in polygons looks bad unless the polygon count is very large. I'm guessing it's because they work better for hardware rendering, which is best at doing very simple operations very many times. (Modern graphics cards may be more flexible, but the industry pretty much settled on polygons a long time ago.)

Suppose you want to write a software rendering game engine (let's say because your target hardware has a fast CPU but a very weak graphics chip). Would polygons still be the way to go?

Conjecture: if you are going to render in software, it's better to play to strengths and use more complex primitives, in particular curved shapes.

Is this correct? If so, are there any known results regarding what kind of curve primitives should be used?

Most software rendering engines dice the parametric primitives to micropolygons, usually on the fly as needed. In essence this reduces the needed complexity to determine intersections. The surface will still look smooth, since each polygon is smaller than a pixel.

This allows for

Data caching.
Discrete geometric derivates.
Displacement is easy to implement.
Same rendering code to be used on most primitive types.

Image 1: Micropolygon render of trimmed NURBS surfaces using Renderman from 2003. No polygon is visible as each sample gets more or less a separate polygon.

But yes it is possible to use iteration method to solve surfaces from implicit formulations. It is typically used in raymarching contexts. The entire render engine could work by doing this its essentially a tradeof between memory and computation.


Ray Tracing with Cones: coverage, overlapping and abutting triangles In his classic paper Ray Tracing with Cones, John Amanatides describes a variation on classical ray tracing. By extending the concept of a ray by an aperture angle, making it a cone, aliasing effects (including those originating from too few Monte Carlo samples) can be reduced.

During cone-triangle intersection, a scalar coverage value is calculated. This value represents the fraction of the cone that is covered by the triangle. If it is less than $1$, it means that the triangle doesn't fully cover the cone. Further tests are required. Without the usage of more advanced techniques however, we only know how much of the cone is covered, but not which parts.

Amanatides states:

  Since at present only the fractional coverage value is used in mixing
  the contributions from the various objects, overlapping surfaces will
  be calculated correctly but abutting surfaces will not.

This does not make sense to me. From my point of view it is the other way around. Let's take an example: We have two abutting triangles, a green and a blue one, each of which covers exactly 50% of our cone. They are at the same distance from the viewer.

The green triangle is tested first. It has a coverage value of 0.5, so the blue triangle is tested next. With the blue one's coverage value of 0.5 our cone is fully covered, so we're done and end up with a 50:50 green-blue mixture. Great!

Now imagine that we kill the blue triangle and add a red one some distance behind the green one - overlapping. Greeny gives us a coverage value of 0.5 again. Since we don't have the blue one to test anymore we look further down the cone and soon find the red one. This too returns some coverage value greater than 0, which it shouldn't because it is behind the green one.

So, from this I conclude that abutting triangles work fine, while overlapping triangles would need some more magic like coverage masks to be correct. This is the opposite of what Amanatides says. Did I misunderstand something or is this a slip in the paper?

I did implement a ray tracer based on Amantides' work but, as that was  years ago, my memory of the paper is a little rusty.

However, ignoring this particular case, in general when it comes to working with fractional coverage e.g. Alpha compositing, (see "A over B") my understanding is that the usual assumption is that the items being composited are uncorrelated.

Thus if A with X% coverage is on top of B with Y% coverage and C in the background, then it's assumed that one will see
    X%*A + (100-X%)*Y% * B + (100-X%)(100-Y%)*C

Does that make sense? Obviously this will give "leaks" in the case where A and B are strongly correlated.

I think I might have put a small bit mask on the rays to avoid these problems, but it was a very long time ago.


What framework shoud I use to implement the RAISR upscaling algorithm on the GPU? I recently ran across the paper RAISR: Rapid and Accurate Image Super Resolution and thought it seemed like an interesting technique for scaling up images. However, I haven't seen any implementation of it, and I would like to see what it would be like to use it for scaling up a video in realtime, if that is possible. So I'm thinking of implementing my own version of it, but I will need to use GPU acceleration in some way if I want to at least give it a chance to run in realtime. Ultimately, I would like to see if I could successfully implemented this in an open source media player.

So, what library should I use to implement this algorithm? DirectX and CUDA are out of the question, since I want to make it run under any (major) OS and on any graphics card (well, on most of the popular ones, at least), so I'm thinking that my only options are

OpenGL (ES) + GLSL (ES), and
OpenCL.

(Vulkan also exists but I don't know whether this API is mature enough yet.)

If I choose OpenGL + GLSL, is this approach flexible enough to allow this algorithm to be implemented? Are there algorithms that cannot be implemented this way and have to be implemented in OpenCL, and if so, how do I know if my algorithm is one of them?

If I on the other hand choose OpenCL, will it be possible for other people to use my implementation without installing OpenCL on their computer?

Are there algorithms that cannot be implemented this way and have to be implemented in OpenCL, and if so, how do I know if my algorithm is one of them?

Yes. Generally scatter-gather algorithms work poorly in OpenGL as the data locality is poor. When you have to grab bits from all over the input image for each output pixel, it's going to be slow or nearly impossible to implement in a typical GL shader. Likewise, when a calculation on an input pixel results in data that needs to be spread out to several output pixels, it's a poor fit.

It's hard to tell just from the abstract, but it looks to me like this wouldn't be a great fit for a typical OpenGL shader. I think you'll end up needing to sample an area of the input, classify it, then sample from one or more of a large set of small exemplars.

I mainly work on iOS and macOS which both come with OpenGL and OpenCL. Not sure how Windows and Linux handle that.


What are the most efficient ways to convert between f32 and smaller float formats on the CPU? Open GL and other graphics APIs support floating point formats smaller than 32 bits (e.g. see https://www.opengl.org/wiki/Small_Float_Formats). While GPUs seem to handle these formats natively, CPUs have limited support (x86 architecture chips can convert to/from 16 bit floats using the AVX instructions). It is sometimes convenient to pack/unpack these formats on the CPU , but it is not obvious how to do so efficiently.

What are the fastest techniques to convert between 32 bit floating point and the small float formats on the CPU? Are libraries or sample implementations available?

You definitely want to check out meticulous half to float and float to half implementations by Fabian Giesen. I don't think there is anything faster than those:
https://gist.github.com/rygorous/2144712
https://gist.github.com/rygorous/2156668

For other formats you will find an implementation in DirectXMath (part of WindowsSDK), Mesa and similar libraries. For example, R11G11B10_Float conversions are implemented by DirectXMath in XMFLOAT3PK and by Mesa in u_format_other.c.


Modern way of light sources in OpenGL? In older versions of OpenGL there were things like:

glLight(...)
gl_LightSourceParameters

The modern way of doing lighting is passing light source parameters manually to the shader in vao-s? Or uniforms?

Why are these things decrepated?

And if I have 100 light sources in the scene, I obviously can't compute lighting with all of them. So somehow in a specific object's shading I have to only care about the light sources which are important for it.

I think that I should move for example only 8 light sources to the shaders, based on their "strength".

But isn't this too complicated to call every frame?

How should I handle lots of light sources (in the scene)?

/I read about deferred lighting, but is that the only way?/

Why are they deprecated?

Those functions are deprecated because the OpenGL API moved in favor of a programmable pipeline in contrast with the old fixed pipeline. The programmable pipeline allows the flexibility necessary to enable a wide variety of effects and solutions that before were difficult or not possible at all to implement. You can still access those functions, as they are necessary to support older systems, but besides that I don't see any reason not to use the programmable pipeline.

How to pass parameters

If we are talking about parameters of lights (such as their position or intensity), you can pass those using uniforms. Particularly, you should look at Uniform Buffer Objects.

It would look like this in a shader:

struct Light {
   int type;
   vec3 diffuse;
   vec3 position;
   bool on;
   ...
};

#define MAX_NUM_LIGHTS 8

uniform lightBlock {
    Light lights[MAX_NUM_LIGHTS];
};

Note that this already gives you a way of passing the lights that you want. Make your shaders expect an uniform buffer with 8 lights, copy those into an array in your client code and send them using the API. For more info on how to do this using the API functions, check the link above about UBOs and this.

Is it too complicated to call every frame?

Depends on your scene and the criteria for how to choose the lights that you want to pass to the shader. If you are worried about sending the data of 8 lights through uniforms each frame, then don't, that is the least of your worries. The problem lies in the selection process of the lights and the complexity of your scene/shading model, as the cost increases with the number of lights.

Forward Renderer: If your scene and shading models are simple, maybe 8 lights are okay. If you implement modern shading models, 8 lights might be a bit too much. The point is that you will have to test it with the scene and shading models and find a compromise between all of that. Maybe pass 4 instead of 8 lights, maybe you should simplify shading models, etc.
Deferred Renderer: A deferred renderer does help with multiple light rendering due to the way it is designed. However, deferred rendering may introduce other difficulties as it is, in my opinion, a more complex system. Also, transparency and other aspects may be a pain to deal with.
Other considerations: Keep in mind that there are possible optimizations. It may be possible to optimize specific lights. Maybe not all of them need to be dynamic: think of light maps and other image based lighting. You should consider if your scene can apply any of this.

In summary, choosing between deferred or forward is a big decision. Carefully think about your scene requirements first and devise a strategy for light selection. Take into account the complexity of your shading models. For optimization, you can combine several techniques, such as simplifying some lights that are far away from the camera and only pass the relevant ones. When you have a framework for testing, find a compromise between all of this.