How to link Intel Open Image Denoise in Qt Creator and use it to denoise noisy Monte Carlo path tracing renderings? I've been trying to link my the Intel Open Image Denoise libraries for about a week and still have no luck as of now. This could partly be because I am also novice in C++ stuff. So I need some help in getting it to work.
Here's what I've done so far: after compiling OIDN I moved the oidn directory to the directory where my path tracer code is and did the following in my code (C++11) which I'm working on in Qt Creator.
#include "oidn/include/OpenImageDenoise/oidn.hpp"
.
.
.
oidn::DeviceRef device = oidn::newDevice(); // this is where things break
device.commit();

Note that doing #include "oidn/include/OpenImageDenoise/oidn.hpp" seems to work fine because autocomplete works for oidn and shows its members. However I get the following errors which seems to be due to not linking oidn libraries.

I tried linking the libraries via LIBS += -L"oidn/build/" or LIBS += -L"oidn/build/libOpenImageDenoise.so" (as shown in the image) but I get the same errors. I also appended -loidn but got an error saying "cannot find -loidn". I am now pretty confused on what I'm supposed to do. I'm not even sure if the errors are due to not properly linking libraries because I see people generally link libraries in C++ the same way I am trying to do it here. I would really appreciate if someone can help me figure out what I need to do to get OIDN work.

All the previous answers/comments were correct. The weird thing about my case was the symlinks of the libraries in the build directory were not valid and were not pointing to the right files. The output of ls -lh /home/Warrior/Desktop/pathtracer/oidn/build was as follow:
total 160K
drwxrwxr-x 4 amir amir 4.0K Feb  4 00:09 apps
-rw-rw-r-- 1 amir amir  19K Feb  4 00:09 CMakeCache.txt
drwxrwxr-x 7 amir amir 4.0K Feb  4 00:09 CMakeFiles
-rw-rw-r-- 1 amir amir 6.5K Feb  4 00:09 cmake_install.cmake
drwxrwxr-x 3 amir amir 4.0K Feb  4 00:09 common
-rw-r--r-- 1 amir amir 3.6K Feb  4 00:09 CPackConfig.cmake
-rw-r--r-- 1 amir amir 4.0K Feb  4 00:09 CPackSourceConfig.cmake
-rw-rw-r-- 1 amir amir 103K Feb  4 00:09 Makefile
drwxrwxr-x 3 amir amir 4.0K Feb  4 00:08 mkl-dnn
-rw-r--r-- 1 amir amir 1.6K Feb  4 00:09 OpenImageDenoiseConfigVersion.cmake

But when I would go to the directory I would see the following:

What I did was instead of cloning the oidn repo somewhere else and building it, I cloned the oidn repo in /home/Warrior/Desktop/pathtracer/ and built it there. After that I could see the correct file names/symlinks via ls -lh as follow. Now my program compiles and things work fine. I just had to use LIBS += -L"oidn/build/" -lOpenImageDenoise when compiling my code:
total 51M
drwxrwxr-x 4 amir amir 4.0K Feb  9 10:14 apps
-rw-rw-r-- 1 amir amir  19K Feb  9 10:14 CMakeCache.txt
drwxrwxr-x 7 amir amir 4.0K Feb  9 10:19 CMakeFiles
-rw-rw-r-- 1 amir amir 5.6K Feb  9 10:14 cmake_install.cmake
-rw-rw-r-- 1 amir amir 2.5K Feb  9 10:17 color_ispc_avx2.h
-rw-rw-r-- 1 amir amir 2.5K Feb  9 10:17 color_ispc_avx512skx.h
-rw-rw-r-- 1 amir amir 3.1K Feb  9 10:17 color_ispc.h
-rw-rw-r-- 1 amir amir 2.5K Feb  9 10:17 color_ispc_sse4.h
drwxrwxr-x 3 amir amir 4.0K Feb  9 10:14 common
-rw-r--r-- 1 amir amir 3.3K Feb  9 10:14 CPackConfig.cmake
-rw-r--r-- 1 amir amir 3.7K Feb  9 10:14 CPackSourceConfig.cmake
-rw-rw-r-- 1 amir amir 2.6K Feb  9 10:17 input_reorder_ispc_avx2.h
-rw-rw-r-- 1 amir amir 2.6K Feb  9 10:17 input_reorder_ispc_avx512skx.h
-rw-rw-r-- 1 amir amir 3.1K Feb  9 10:17 input_reorder_ispc.h
-rw-rw-r-- 1 amir amir 2.6K Feb  9 10:17 input_reorder_ispc_sse4.h
-rw-rw-r-- 1 amir amir  41K Feb  9 10:14 libcommon.a
-rw-rw-r-- 1 amir amir  16M Feb  9 10:17 libdnnl.a
lrwxrwxrwx 1 amir amir   24 Feb  9 10:19 libOpenImageDenoise.so -> libOpenImageDenoise.so.0
lrwxrwxrwx 1 amir amir   28 Feb  9 10:19 libOpenImageDenoise.so.0 -> libOpenImageDenoise.so.1.3.0
-rwxrwxr-x 1 amir amir  34M Feb  9 10:19 libOpenImageDenoise.so.1.3.0
-rw-rw-r-- 1 amir amir  37K Feb  9 10:19 libutils.a
drwxrwxr-x 2 amir amir 4.0K Feb  9 10:17 local__core
-rw-rw-r-- 1 amir amir 104K Feb  9 10:14 Makefile
drwxrwxr-x 3 amir amir 4.0K Feb  9 10:14 mkl-dnn
-rwxrwxr-x 1 amir amir 181K Feb  9 10:19 oidnBenchmark
-rwxrwxr-x 1 amir amir  59K Feb  9 10:19 oidnDenoise
-rwxrwxr-x 1 amir amir 882K Feb  9 10:19 oidnTest
-rw-r--r-- 1 amir amir 1.6K Feb  9 10:14 OpenImageDenoiseConfigVersion.cmake
-rw-rw-r-- 1 amir amir 1.5K Feb  9 10:17 output_copy_ispc_avx2.h
-rw-rw-r-- 1 amir amir 1.5K Feb  9 10:17 output_copy_ispc_avx512skx.h
-rw-rw-r-- 1 amir amir 1.9K Feb  9 10:17 output_copy_ispc.h
-rw-rw-r-- 1 amir amir 1.5K Feb  9 10:17 output_copy_ispc_sse4.h
-rw-rw-r-- 1 amir amir 2.5K Feb  9 10:17 output_reorder_ispc_avx2.h
-rw-rw-r-- 1 amir amir 2.5K Feb  9 10:17 output_reorder_ispc_avx512skx.h
-rw-rw-r-- 1 amir amir 3.1K Feb  9 10:17 output_reorder_ispc.h
-rw-rw-r-- 1 amir amir 2.5K Feb  9 10:17 output_reorder_ispc_sse4.h
-rw-rw-r-- 1 amir amir 1.5K Feb  9 10:17 upsample_ispc_avx2.h
-rw-rw-r-- 1 amir amir 1.5K Feb  9 10:17 upsample_ispc_avx512skx.h
-rw-rw-r-- 1 amir amir 1.8K Feb  9 10:17 upsample_ispc.h
-rw-rw-r-- 1 amir amir 1.5K Feb  9 10:17 upsample_ispc_sse4.h
drwxrwxr-x 2 amir amir 4.0K Feb  9 10:17 weights


What explains the high specularity of metals? From my understanding, the specular color usually refers to the amount of light that is reflected when the surface is lit at normal incidence, and is noted $F_0$ or $R_0$. Moreover, for non metal materials, this value is calculated from the index of refraction of the material $n$ with the formula deduced from the Fresnel equations (in which 1 is the index of refraction of air or void):
$$F_0 = \frac{(n - 1)^2}{(n + 1)^2} $$

According to this list of refractive indices on Wikipedia:

Solid materials typically have $n$ between 1.46 (fused quartz) and 2.69 (Moissanite). That would mean an $F_0$ between 0.03 and 0.21.
Liquids typically have $n$ between 1.33 (water) and 1.63 (carbon disulfide). That would mean an $F_0$ between 0.02 and 0.057, if I am not mistaken.
Gases typically have $n \approx 1$, so I guess we can safely assume an $F_0$ of 0.

All these values are very low; even crystals with high refractive indices like diamond ($F_0=0.17$) and moissanite ($F_0=0.21$) hardly exceed 20%. Yet most metals have $F_0$ values above 50%. Moreover, I have read multiple times that the formula mentioned above doesn't apply for metals (which can be easily confirmed by trying to use it and see completely wrong results), but I haven't found any further explanation.

What phenomenon explains this difference? How can I calculate $F_0$ for a metal (in particular if the medium it is in contact with has an IoR different than 1, like water)?

Warning: I am not a physicist.

As Dan Hulme already explained, light can't travel through metals, so dealing with IOR is a lot more... complex. I will answer why that happens and how to calculate the reflection coefficient.

Explanation: Metals are filled with free electrons. Those electrons react to external fields and reposition until electrostatic equilibrium is met (the electric field is zero inside a conductor in electrostatic equilibrium). When electromagnetic waves hit a metallic surface, the free electrons move until the field that they create cancels the field of the incoming wave. Those electrons grouped together radiate a wave going out nearly the same as the one that hit the surface (i.e. with very low attenuation). How much is attenuated depends on the material properties.

From this explanation it is clear that conductivity is a key part of the high reflection coefficient on metals.

Math-wise, what you are missing is the complex index of refraction. On good conductors, such as metals, the complex term of the IOR is relevant and key for explaining this phenomena.

Practically, in rendering, achieving good metal parameters is more visual based. Artists adjust to their preference until it looks believable. Often you see a metalness parameter with specific handling for materials marked as metal.

Involved answer:

The complex index of refraction can be seen if we use Ohm's Law $J = \sigma \vec{E}$, which holds for conductors, on the Ampère-Maxwell equation using sinusoidal waves $\vec{E} = e^{i\omega t}$:

$$
\vec{\nabla} \times \vec{H} = \sigma\vec{E} + \frac{\partial \vec{D}}{\partial t} = \sigma \vec{E} + i\omega \epsilon \vec{E}
$$
$$
 = i\omega \left( \epsilon - i \frac{\sigma}{\omega} \right)\vec{E} = i \omega \epsilon_m\vec{E}
$$

Note how we can interpret that whole term as a complex permittitivity $\epsilon_m$ and that $\sigma$ is the conductivity of the material.

This affects the IOR, as its definition is given by:

$$
  n' = \sqrt{\frac{\epsilon_m}{\epsilon_0}} = \sqrt{\frac{\left(\epsilon - i \sigma / \omega\right)}{\epsilon_0}} = n_{\text{real}} + in_{\text{img}}
$$

This shows how $n'$ can be complex. Also, note how very good conductors have a relevant complex term, as $\sigma \gg \epsilon_0 \omega$. Since it would take a lot, I will skip some steps with a reference, page 27: it can be shown that, since $\sigma \gg \epsilon_0\omega$, (we are dealing with $\omega$ of the visible spectrum):
$$
  n_{\text{real}} \approx n_{\text{img}}
$$

and reflection from metals with normal incidence, from a medium with IOR $n$, given that $n' \gg n$:

$$
  R = \frac{(n_{\text{real}} - n)^2 + n_{\text{img}}^2}{(n_{\text{real}} + n)^2 + n_{\text{img}}^2} \approx 1
$$

Agreeing that a good conductor is, in general, a good reflector.

The famous Introduction to Electrodynamics from Griffiths, pages 392-398, explains this and a lot more in a similar fashion.


OpenGL : How to rotate an object around viewing space x-axis I have an object(cow).

This cow was translated and rotated(with arbitrary axis) in the modeling space for several times.

And I can't know how much cow is translated or rotated.

Only I can obtain is this cow's current matrix and it's inverse.

In this circumstance, how can I rotate the cow around the x-axis in the viewing space?

(center of rotation is at the center of modeling space)

A rotation around the x-axis is achieved by this matrix
$$
\mathtt{T}_\mathrm{rot X} = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0& \cos \alpha
            & -\sin \alpha& 0 \\ 0 & \sin \alpha & \cos \alpha & 0 \\ 0 & 0 & 0 &1 \\ \end{bmatrix}
$$

In order to rotate around the x-axis in the viewing space, one needs to think about when to apply the above transformation in the chain of transformations that is applied to the 3D model vertices $\mathbf{X}_{\mathrm{3d}}$ in the employed rendering pipeline.

A typical transformation chain is:

$$
\mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{camera\ transform}}\,
\mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}
$$

Let's split the camera tranformation into its rotational and translational part:
$$
\mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\underbrace{\mathtt{T}_{\mathrm{cam\ trans}}\,\mathtt{T}_{\mathrm{cam\ rot}}}_{\mathtt{T}_{\mathrm{camera\ transform}}}\,
\mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}
$$

Then we need to apply the x-rotation in-between the camera rotation and translation, because we want to have the rotation of the camera applied beforehand, but not the camera's translation:

$$
\mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\underbrace{\mathtt{T}_{\mathrm{cam\ trans}}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}}_{\mathtt{T}_{\mathrm{new\ camera\ transform}}}\,
\mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}
$$

This would achieve the desired rotation, but typically we do not want to change the camera transformation but the the mesh transformation. This is achieved by
$$
\mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{cam\ trans}}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}\,
\mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}\\
\Leftrightarrow \mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{cam\ trans}}\,(\mathtt{T}_{\mathrm{cam\ rot}}\,\mathtt{T}_{\mathrm{cam\ rot}}^{-1})\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}\,
\mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}\\
\Leftrightarrow \mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{camera\ transform}}\,\underbrace{\mathtt{T}_{\mathrm{cam\ rot}}^{-1}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}\,
\mathtt{T}_{\mathrm{mesh\ transform}}}_{\mathtt{T}_{\mathrm{new\ mesh\ transform}}}\,\mathbf{X}_{\mathrm{3d}}
$$
Therefore, the short answer to your question is:

Left-multiply the current transformation matrix of your model with $\mathtt{T}_{\mathrm{cam\ rot}}^{-1}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}$.

I have generated an "cow" example scene in the GSN Composer that can be used for interactive testing in a WebGL-capable browser:

Example: CowQuestion | GSN Composer


Why do most photorealistic renderers have similar material test scenes? I notice that most photorealistic renderers have very similar material test scenes. Here are a few examples:

Blender

Mitsuba

Vray

My question is: why is this specific model / style chosen to test materials? Does it have some advantage over teapots, spheres, and suzannes? Are there other common scenes for material testing?

While teapot, spheres and trusty Suzanne are not per se bad test scenes for materials, here are some things to consider. You can apply them to said examples to forge your own opinion.

First and foremost, the viewer needs to be able to examine the behavior of the BRDF. Since it is dependent on both the direction of incident light and the direction from surface to viewer, you need to have as many combinations as possible in your test scene. A spheroid nicely covers all possible viewer angles.

Notice that all three scenes feature some sharper edges or creases too. Differences in reflective behavior across a smooth surface (like a sphere) may be hard to conceive, so these edges put emphasis on this.

Since the observed brightness and color is a combination of the material (i.e. the BRDF) and the lighting environment, it may be hard to tell one from another. The Vray test scene takes care of this. The 100% white 'core' and the 25% black floor help you get an idea of what the lighting in the scene is like.

Blender and Vray both feature a grid that helps you estimate the total size of the scene. This helps with evaluating grain or surface texture.

Here is a side-by-side comparison of a cube, notched sphere and Suzanne for a glossy test material with a slight normal map (rendered in Blender with Cycles, lit with the Dutch Skies 360 - Free 002 HDR):

The cube does a pretty good job on depicting the normal map. The sphere, as expected, is reasonably good for normals and gloss. Suzanne, arguably, makes both a bit harder to perceive than the spheroid.


Do stencil tests use any acceleration techniques? Stencil bits are often used for optimizations, for instance having full screen lighting passes only affect certain pixels.

Does the stencil test mechanism use any acceleration structures of any kind (like maybe dividing the screen into a coarser grid?) or do the perf benefits come solely from having a pre-pixel shader test that can early out waves of shaders?

I don't have a definite authoritive answer for you but I'm very confident that the answer is "yes". Here's why:
When using framebuffer objects, stencil attachments are usually combined with the depth attachment in a single texture (GL_DEPTH24_STENCIL8 or GL_DEPTH32F_STENCIL8). In fact, until recently, a combined depth and stencil was the only way to get a stencil buffer. It is now possible to create a stencil only attachment for an FBO.
This OpenGL wiki article on FBOs states (emphasis mine):

These color formats can be combined with a depth attachment with any of the required depth formats. Stencil attachments can also be used, again with the required stencil formats, as well as the combined depth/stencil formats. However, implementations are only required to support both a depth and stencil attachment simultaneously if both attachments refer to the same image.
This means that if you want stencil with no depth, you can use one of the required stencil formats. If you want depth with no stencil, you can use one of the required depth formats. But if you want depth and stencil, you must use a depth/stencil format and the same image in that texture must be attached to both the depth and stencil attachments.

All this strongly hints towards an accelerated storage format for stencils similar to the one used by depth buffers.
Also, empirically, over the years many rendering algorithms (e.g. stencil shadows) using the stencil buffer would have been prohibitively slow if stencil buffers did not use some form of accelerated structure.
The NV_packed_depth_stencil extension (2001) spec also provides interesting insight:

Many OpenGL implementations have chosen to interleave the depth and stencil buffers into one buffer, often with 24 bits of depth precision and 8 bits of stencil data.

and (emphasis mine)

Should the depth/stencil format support other standard types, like GL_FLOAT or GL_UNSIGNED_INT?
RESOLVED: No, this extension is designed to be minimalist. Supporting more types gains little because the new types will just require data format conversions.  Our goal is to match the native format of the buffer, not add broad new classes of functionality.


Force int array to compact memory I have following structure which i use as an uniform buffer object.

struct GlobalShaderData
{
    #define CLUSTER_X_MAX 16
    #define CLUSTER_Y_MAX 8
    #define CLUSTER_Z_MAX 8
    #define CLUSTER_MAX_LIGHTS 4
    int pointLightIndices[CLUSTER_X_MAX * CLUSTER_Y_MAX * CLUSTER_Z_MAX * CLUSTER_MAX_LIGHTS];
    int pointLightsPerTile[CLUSTER_X_MAX * CLUSTER_Y_MAX * CLUSTER_Z_MAX];
};
GlobalShaderData globalShaderData;

The total size of the first int array should be 4 bytes * 16 * 8 * 8 * 4 = 16384 bytes, thus the maximum ubo memory is not exceeded.

However, when compiling the shaders I receive an "out of resource" error message.

I guess the integer actually takes 16 bytes on the glsl side to align with a vec4.
Because if we calculate with 16 bytes * 16 * 8 * 8 * 4 = 65536 bytes, which just exceedes the maximum capacity.

What are my options to have an smaller integer array on glsl side.

(I know that I could use other storage buffers but thats not relevant to the question.)

Assuming you use this structure within an actual UBO (which your example code doesn't do), just use an array of ivec4 and two indices.

const ivec3 cluster_max = ivec3(16, 8, 8);
const int cluster_max_lights = 4;

const int plix_count = cluster_max.x * cluster_max.y * cluster_max.z * cluster_max_lights;
const int plix_index_size = pl_count / 4;

const int plpt_count = cluster_max.x * cluster_max.y * cluster_max.z;
const int plpt_index_size = plpt_count / 4;

struct GlobalShaderData
{
    ivec4 pointLightIndices[plix_index_size];
    ivec4 pointLightsPerTile[plpt_index_size];
};

layout(std140) uniform shaderData
{
    GlobalShaderData globalShaderData;
};

To access a value from the arrays, you will have to divide the index by 4 to get the ivec4 index, then take the original index mod 4 and use that to index the ivec4.
To get the


Using multiply and accumulate of 4x4 matrices for ray-triangle intersection tests on GPU Is it possible to gain performance boost using new 4x4 MAD from NVIDIA'a tensor cores for ray-triangle intersection tests?

Really there are two questions:

Is it possible to modify some of the ray-triangle algorithms (say, Möller-Trumbore algorithm) to base it on 4x4 matrix MAD.
Would it be a performance improvement?

Yes, it is possible.  For one however, Möller-Trumbore is not the only algorithm out there for ray triangle intersection, there exist others.

However all of these rely on linear algebra vector multiplications and matrix operations.  These can be combined into 4x4 tensor core operations (and each tensor core can do 4x4x4, but there are 8 cuda cores to every tensor core).   You can compute multiple intersections in Möller-Trumbore at a time with tensor cores.  You could say, combine multiple cross products or dot products into a single matrix operation.

The catches?  Well first, the precision of tensor cores matrix multiplication is 16bit float.  But don't fret, given decent spatial subdivision of, say, a raytraced environment, that precision should only matter per spatial block/voxel/subdivision which is necessary in ray-tracing in order to not test all triangles in the scene. 16 bit for a 10 km area might be a big deal in terms of poor position precision, but for a 2x2 meter area it isn't (mantissa alone will handle 1024 different values). Note the addition part is 32bit float.

Second, you have to move data around in such a way that tensor cores can actually access it, which, because it is unlikely that any of us on this entire site have direct access to a machine with tensorcores, has a yet unknown performance cost.

Third, I'm not entirely sure its possible to do both tensor core calculations and scalar calculations at the same time.  This could end up slowing many tasks to the point that any advantage tensor cores gave becomes negligible.  But I could be wrong and we can actually hide things like latency for tensor core memory moves with multiple floating point and tensor core instructions interleaved.


Layer rendering in Vulkan I am writing a Vulkan video manipulation engine which has the following layers:

Foreground (2D)
Scene (3D)
Background (2D)

Note that each layer may have multiple objects, some of which may be transparent. For 2D objects (foreground and background), I want to preserve the relative between objects. Therefore, I thought about the following technique, in order to execute the FS the least amount of times possible:

Draw the opaque foreground objects backwards, writing 0x02 at the stencil buffer.
Draw the opaque scene objects backwards (sorted by average depth), writing and testing 0x01 at the stencil buffer and writing and testing at the depth buffer.
Draw the opaque background objects backwards, writing and testing 0x00 at the stencil buffer.
Draw the transparent background forwards, writing and testing 0x00 at the stencil buffer.
Draw the transparent scene objects forwards (sorted by average depth), writing and testing 0x01 at the stencil buffer and writing and testing at the depth buffer.
Draw the transparent foreground forwards, writing and testing 0x02 at the stencil buffer.

However, this technique does not work as specified, as the relative order between transparent and opaque objects at the background and foreground won't be preserved. This other technique would satisfy the condition (regardless of cherry picking OIT issues) although it is not efficient:

Draw all the background objects forwards
Draw the opaque scene objects backwards (sorted by average depth), writing and testing at the depth buffer.
Draw the transparent scene objects forwards (sorted by average depth), writing and testing at the depth buffer.
Draw all the foreground objects forwards

I would like to know if there is any technique with Vulkan that lets me implement the first algorithm without the mentioned problems. I've thought about a secondary depth buffer in which the relative order between 2D layers gets written (write and test a constant), but this leads me to a new concern:
According to some posts, if gl_FragDepth gets written from a shader (remember that I pretend to write a constant value from an uniform buffer to the depth value), fragment shader gets fully executed, which throws away the purpose of this optimization.

I think you may be overcomplicating things with your idea of using the stencil. In 2D graphics layers, we usually do not need to use the depth buffer to resolve depth order between 2D elements (their depth order is determined by draw order instead). Therefore, we can reserve the nearest depth value for all the foreground elements, and the farthest depth value for all background elements.
This would work as follows, assuming depth 0.0 is the far plane and depth 1.0 is the near plane.

Render opaque foreground elements in front-to-back order, using depth test VK_COMPARE_OP_GREATER, and arranging in the vertex shader for all elements to be placed at depth 1.0.
Render opaque scene objects roughly front-to-back.
Render opaque background elements in front-to-back order, placing them at depth 0.0.
Render transparent background elements back-to-front, now using depth test VK_COMPARE_OP_GREATER_OR_EQUAL, disabling depth writes, and placing them at depth 0.0.
Render transparent scene objects back-to-front.
Render transparent foreground elements back-to-front, placing them at depth 1.0.

The switch from GREATER to GREATER_OR_EQUAL is important here. Since the opaque elements are going front-to-back, you want to exclude the "equal" case, since you want later elements to be occluded by earlier elements. Then, when coming back-to-front for the transparents, you want to include the "equal" so that later elements will correctly render on top of earlier ones in the foreground/background layers.
Edit: Now that I'm thinking about this more, this may not work since transparent elements in foreground/background won't be able to be occluded by opaque elements in the same layer. This could be addressed by pre-clipping the transparent element geometry, or if this is important to support on the GPU, it could be addressed by rendering opaque elements depth-only (no fragment shader) in the first passes and then rendering all elements, both opaque and transparent, back-to-front in the later passes.


How can I account for specular interactions in shadow rays? I've been working through PBRT, implementing my own version of it in Rust. Right now I'm working on shadow rays, and it looks like pbrt doesn't do anything to account for specular behavior, and treats materials like perfect glass as opaque with respect to shadows. I decided to try to implement a version that accounts for this behavior, but I don't see a difference visually. Ignoring the bug that causes some of my threads to get stuck, the top one is my attempt at accounting for specular behavior and the bottom one is the 'normal' implementation that casts a single ray to see if it hits the light or not.

I can't see any difference in how the ground looks, and I don't believe that's how it should look for a perfectly specular glass model. My implementation is as follows  (it's in Rust, so I've commented to get the general idea across):
pub fn occlusion_intersects(&self, ray: &Ray, tmin: f64, tmax: f64, depth: i32) -> Option<HitRecord> {
    // find intersection with scene
    let ans = self.intersects(ray, tmin, tmax);
    // if no intersection was found, return None
    if ans.is_none() {
        return None
    }
    let mut ans = ans.unwrap(); // rust stuff, ignore this
    // compute the bsdf for the material
    Material::compute_scattering_default(&mut ans);
    // we only want to specular transmission or reflection
    let bsdf_type = BSDF_TRANSMISSION | BSDF_REFLECTION | BSDF_SPECULAR;
    // sample_f returns a new outgoing direction according to a bsdf that matches
    // one of the types above, or (0, 0, 0) if no matching bsdfs.
    let (wi, .. ) = ans.bsdf.sample_f(&ans.wo, bsdf_type);
    // if there was no specular bsdf or we've recursed too many times:
    if (wi == util::zero_vector() || depth == MAX_DEPTH {
        // then return the current intersection to check if it
        // hit the expected light object
        return Some(ans);
    }
    // otherwise make a new ray going in the new specular direction
    let new_ray = ans.spawn_ray(&wi);
    // and return that answer
    return self.occlusion_intersects(&new_ray, tmin, tmax, depth + 1);
}

And for reference, the default implementation would be:
pub fn occlusion_intersects(&self, ray: &Ray, tmin: f64, tmax: f64, depth: i32) -> Option<HitRecord> {
    // return the first intersection, ignore all specular behavior
    return self.intersects(ray, tmin, tmax);
}

Is there something wrong with my implementation? I kind of guessed as to how it would work, but the logic written in the comments makes sense to me.

Shadow rays that hit a glass surface or similar are stopped because there's no direct path from the light source to the shading point—there's a piece of glass in the way.

for perfectly specular transmission/reflection, you can continue tracing the shadow ray because there's only one possible outgoing vector for a given incoming vector

That won't work, because the transmission/reflection will change the direction of the ray. The shadow ray was presumably created by pointing it at a light source, but after transmission/reflection, it's most likely not going to hit the light source anymore.
The idea of firing rays directly at light sources can be seen as a sampling strategy designed to exploit knowledge of which directions light is most likely to come from, and put your rays into those directions. If you're receiving light via a specular interaction rather than directly from the source, you don't know anymore which directions you should sample in.
I imagine for simple cases like a glass sphere, etc, you could algebraically solve for a light path that would refract toward a chosen point on a light source. Then you could trace shadow rays along that path. But when you get into geometry as complex as the dragon, there's no realistic way to calculate which directions the light is going to come in from.
When specular surfaces like glass or metal focus light onto a diffuse surface, that produces a caustic. Caustics are a notorious pain point for vanilla path tracing due to this problem—there is no way to know where to concentrate your samples to make your rays hit light sources, so the best you can do is fire rays in all directions (or sample from the BRDF) and hope that enough of them will find their way to a light source. Technically, path tracing can render caustics correctly, but it takes an extremely large number of samples to do it.
More practically, this issue can be solved by using photon mapping (which was invented precisely to deal with caustics), or some kind of path guiding methods like Metropolis and friends that can "discover" the caustics and dynamically concentrate samples around them.


Why do we have to partition non-convex polygons into convex polygons in a map in Binary Space Partitioning? This article says that binary space partitioning divides the map into convex polygons.

  [...]
  Before the map can be rendered we must perform a number of
  calculations on it. However, once these calculations are performed
  their results can be used many times. This is one of the advantages of
  BSP — once the calculations are performed they do not need to be done
  again, unless the map is changed. BSP only allows ”static” maps, or
  ones that do not move. If a map has any moving parts then they must be
  rendered separately.

  What must be done is to partition, or divide up, the map into convex
  polygons.

Why should non-convex polygons be partitioned into convex polygons? Is there a mathematical basis behind this?

Because with 2 non-overlapping convex polygons you can always say that 1 polygon is closer to a point than another.

If they are not convex then for example with a U surrounding a circle. You cannot say easily which should be drawn first.


Rendering moist/wet materials like human anatomies for surgical simulators Real human anatomies look wet/ moist when operated. I am looking for some good methods for rendering anatomies in a virtual simulation, and hence I have done a literature survey on 'Rendering wet materials' using keywords like:

Rendering wet materials.
Rendering moist surfaces.
Rendering wet surfaces.
etc.

What I observed is, in general, people use ray-tracing methods after modeling surface with a two-layer surface reflection model, for achieving realism for wet surfaces.
What are some good rasterization methods/ approximations for the same keeping in mind real-time performance and unavailability of RTX gfx cards, as well as realism?

The reason why ray-tracing is often used for achieving as-realistic-as-possible wet appearance is their reflective behavior. A thin liquid film on the surface reflects photons like a mirror. If you target real-time rendering, you will have to fake the reflections (or rely on RTX :) There are two ways I can quickly think of: placing reflection probes in the scene using cube-maps or using screen space reflections.
Reflection probes with cube-maps

good for static scenes, you can pre-compute many of them
allow objects outside of the current viewing frustum to be seen in the mirrors
inaccurate due to limited number of probes

There is a great extensive tutorial by Sébastien Lagarde on reflections which continues (what a coincidence) with part B on wet surfaces and there is part C as well. Make sure to check them out!
Screen-space reflections

fine also for dynamic scenes
only objects in the current viewing frustum can be seen in the mirrors
accurate within the frustum

Screen space reflections are explained as part of this awesome series by David Lettier.
Subsurface scattering
Regarding the discussion on subsurface scattering, any organic tissue exposes this property. We just observe it the most on skin or leaves, but the same effect is observed inside the body. Just read section 3.3 of this paper:

One significant casual effect under endoscopic environment is a strong subsurface scattering on the surface of internal organs.

I suggest to use at least some post-processing technique for that.


How to shift color values of a single RGB channel I’m a weaver and textile maker with no prior coding or programming experience. In researching and reading through posts on this and other message boards, I was able to find a lot of useful information, but nothing that I could clearly map on what it is I’m trying to do. I’m hoping someone can steer me in the right direction.

I’m working with a set of 18x18 (324px), 8 bit RGB color PNGs which will ultimately be converted into weave structures. I am interested in implementing a script which will allow me to change the value of all pixels of a particular color channel. I’m seeing a number of posts describing how to change the color values of specific pixels, but little about making global changes to the color channels themselves.

I want to be able to manipulate each color channel separately. For example, I'd like to be able to edit all RED pixels in an image such that all R pixel values increase by, say, 2, while pixels containing no R remain unchanged. Additionally, I want to be able to specify a limit, e.g. if the maximum R value in the image is 178, all R pixel values equaling 178 will remain unchanged, while all pixels with R values less than 178 will increase in value by 2.

I hope to be able to specify the channel (e.g. RED), amount of change (e.g. 2, 1, etc.), type of change (e.g. increase/decrease), and limit (e.g. n = 178, n = 0, n = 255, etc.) as those variables will need to be adjusted to satisfy different color palettes and weave structures.

I've tried achieving this in Photoshop and other graphics editors, but outside of selecting like pixels and adjusting their values manually, I haven't had luck in finding an effective solution for this.

I’m wondering - 1) how feasible this is, 2) what program (e.g. Gimp, ImageMagick, etc.) and/or programming language might provide the most straightforward, expedient solution and 3) if anybody has any tips about how go about writing a script like this, as this is all quite new to me.

I don't have any direct experience doing this so I might be missing an obvious solution or tool. That said, what you describe is in programming terms comparatively easy to achieve. The basic structure of such a custom processing would be:

Open the image file and get access to the array of pixels it contains.
Iterate over all the pixels and inspect/transform however you want.
Write out the resulting modified image in the desired format.

Both Gimp (1, 2) and Photoshop (tutorial) provide scripting or plugins support which will allow you to provide custom logic for step 2 and handle the rest for you. I don't have any experience with either, but from a very quick overview, I recommend that you use Javascript, Python or another scripting language which will be much more accessible as a new programmer than a language like C.

Another option I found outside of an image editor that looks good is the Python Imaging Library. Here is a very relevant other question on SO: https://stackoverflow.com/questions/36468530/changing-pixel-color-value-in-pil

From it, I was able to write the following Python script that does what you asked:

from PIL import Image

img = Image.open("input.png")
pixels = img.load() # create the pixel map

# configuration
channel = 0 # 0 = red, 1 = green, 2 = blue
change_amount = 2 # can be -2 for decrease
limit = 178

for i in range(img.size[0]): # for every pixel (in 2D):
    for j in range(img.size[1]):
        if pixels[i,j][channel] < limit:
            # Only modify pixels below the set limit.
            # The following "strange" code is used because Python tuples can't be modified
            # directly. The tuple is transformed into a (mutable) list and moved back again
            # to a tuple afterwards. You can safely ignore this and concentrate on the
            # second line that actually modifies a channel's value.
            color = list(pixels[i,j])
            # Increase (or decrease for a negative change_amount value) the chosen channel.
            color[channel] += change_amount
            pixels[i,j] = tuple(color)

img.save("out.png")

This is untested code, but it shows how to translate what you want into code. You already described what you wanted in a very programmer-friendly manner so the translation is almost direct. The sky is the limit then as to what you can do with pixels once you load them in a custom program to manipulate them.


Creating glsl library I have different functions for example normalmapping which I use all the time...
Now I thought about making a small glsl library.

My problem:
I don't want to have all functions within the same "namespace".

I would like to have it like:

vec4 lighting::Phong(vec3 normal, vec3 color, vec3 lightPositionRelative)

The thing is, that I have tons of functions and sometimes multiple functions are related to the same thing, for example: A HashTable, AddKey, GetKey, HasKey...

So some kind of namespace or even better a class like structure.
What are the options I have?
How are you guys handle this?

GLSL does not support C++ style namespaces. You might want to do some regex magic before parsing the shader, replacing '::' with '_' or something. You might want to have a preprocessor of sorts - or just forget about it entirely.


Power heuristic in multiple importance sampling Both Mitsuba and Tungsten use the power heuristic to do multiple importance sampling (MIS).

Mitsuba:

inline Float miWeight(Float pdfA, Float pdfB) const {
    pdfA *= pdfA;
    pdfB *= pdfB;
    return pdfA / (pdfA + pdfB);
}

Tungsten:

static inline float powerHeuristic(float pdf0, float pdf1)
{
    return (pdf0*pdf0)/(pdf0*pdf0 + pdf1*pdf1);
}

Advanced Global Illumination gives the balance heuristic:
$$F=\cfrac1N\sum_{i=1}^n\sum_{j=1}^{n_i}\cfrac{f(X_{i,j})}{{\sum\limits_{k=1}^n}\frac{c_k}Np_k(X_{i,j})}$$

The power heuristic may be like this:
$$F=\cfrac1N\sum_{i=1}^n\sum_{j=1}^{n_i}\cfrac{f(X_{i,j})p_i(X_{i,j})}{{\sum\limits_{k=1}^n}\frac{n_k}Np_k^2(X_{i,j})}$$

I know they are all valid estimators.

What's the motivation to use the power heuristic?

What about the cube heuristic?
$$F=\cfrac1N\sum_{i=1}^n\sum_{j=1}^{n_i}\cfrac{f(X_{i,j})p_i^2(X_{i,j})}{{\sum\limits_{k=1}^n}\frac{n_k}Np_k^3(X_{i,j})}$$

PBRT v3 Page 799:

Veach determined it empirically.


Readback Data through Buffers I am pretty new to the dx12 world. As a sample project I am trying to add 2 buffers and store the output in yet another output buffer. As apart of this debugging I am trying to see if the buffers that I am loading actually have the right values.
Thus, once the buffer is loaded, I want to put in a fence and "read back the data uploaded" into the CPU and print out the contents of the buffer.
After a lot of googling I thought i should use the map function in DXD12.

But I am not entirely sure how I should go about doing this. Any thoughts?

Ok, so I made little example of how to do this.

Short version

You need to allocate additional buffer with heap type D3D12_HEAP_TYPE_READBACK and resource state set to D3D12_RESOURCE_STATE_COPY_DEST. Then create command list with type D3D12_COMMAND_LIST_TYPE_COPY. After dispatch compute shader use fence to check if buffer is ready, then copy from this buffer to buffer on D3D12_HEAP_TYPE_READBACK heap. Check again with fence if copying is finished and call Map on this buffer.

I would suggest that you stop reading here and try implement it from description above and check longer version if its still doesn't work. Copy/Past isn't the best way to learn :)

Longer version

First lets create new command list.

ID3D12CommandQueue* copyQueue;
ID3D12CommandAllocator* copyAllocator;
ID3D12GraphicsCommandList* copyList;

D3D12_COMMAND_QUEUE_DESC descCopyQueue = {};
descCopyQueue.Flags = D3D12_COMMAND_QUEUE_FLAG_NONE;
descCopyQueue.Type = D3D12_COMMAND_LIST_TYPE_COPY;
device->CreateCommandQueue(&descCopyQueue, IID_PPV_ARGS(&copyQueue));
device->CreateCommandAllocator(D3D12_COMMAND_LIST_TYPE_COPY, IID_PPV_ARGS(&copyAllocator));
device->CreateCommandList(0, D3D12_COMMAND_LIST_TYPE_COPY, copyAllocator, nullptr, IID_PPV_ARGS(&copyList));

Notice type D3D12_COMMAND_LIST_TYPE_COPY.

Next thing is new buffer.

D3D12_HEAP_PROPERTIES const heapPropertiesReadback =
{
    /*Type*/                    D3D12_HEAP_TYPE_READBACK
    /*CPUPageProperty*/         ,D3D12_CPU_PAGE_PROPERTY_UNKNOWN
    /*MemoryPoolPreference*/    ,D3D12_MEMORY_POOL_UNKNOWN
    /*CreationNodeMask*/        ,0
    /*VisibleNodeMask*/         ,0
};

D3D12_RESOURCE_DESC buffersDesc = {};
buffersDesc.Dimension = D3D12_RESOURCE_DIMENSION_BUFFER;
buffersDesc.Alignment = 0;
buffersDesc.Width = sizeof( unsigned int ) * 5;
buffersDesc.Height = 1;
buffersDesc.DepthOrArraySize = 1;
buffersDesc.MipLevels = 1;
buffersDesc.Format = DXGI_FORMAT_UNKNOWN;
buffersDesc.SampleDesc.Count = 1;
buffersDesc.SampleDesc.Quality = 0;
buffersDesc.Layout = D3D12_TEXTURE_LAYOUT_ROW_MAJOR;
buffersDesc.Flags = D3D12_RESOURCE_FLAG_NONE;

ID3D12Resource* bufferReadback;

device->CreateCommittedResource(&heapPropertiesReadback, D3D12_HEAP_FLAG_NONE, &buffersDesc, D3D12_RESOURCE_STATE_COPY_DEST, nullptr, IID_PPV_ARGS(&bufferReadback));

Remember to set D3D12_RESOURCE_DESC ::Width to size of your buffers.

Then after you Dispatch compute list and call ExecuteCommandLists on compute queue you need to use fence, copy buffer and use fence again.

computeQueue->Signal(fence, fenceValue);
fence->SetEventOnCompletion(fenceValue, fenceEvent);
++fenceValue;
WaitForSingleObject(fenceEvent, INFINITE);

copyList->CopyResource(bufferReadback, bufferB);
copyList->Close();
copyQueue->ExecuteCommandLists(1, (ID3D12CommandList**)(&copyList));

copyQueue->Signal(fence, fenceValue);
fence->SetEventOnCompletion(fenceValue, fenceEvent);
++fenceValue;
WaitForSingleObject(fenceEvent, INFINITE);

After this you can call Map on resource.

unsigned int* data;
bufferReadback->Map(0, nullptr, reinterpret_cast<void**>(&data));

I hope this will help, let me know if you have any questions.


How 3D painting software works I am new to graphics programming and i'm learning opengl. I managed to make a really simple renderer where i can import models display theirs textures with all lighting. My goal is to make a simple painting software like Substance painter or Mari but much much simpler and i don't have any idea on how they work ? I googled but couldn't find a good answer.
Thanks.

We had some related questions lately, so you might find them useful:

Paint pixels on the image around the cursor for painting application
How are textures projected onto 3d models in texture painting applications
Get normalized device coordinate of an image pixel

I can't tell you how the named applications do it, since I don't have access to the code, but painting onto a 3d model could be achieved as follows:
When you click the "paint" button, you cast a ray from the camera position into your 3d scene. For perspective projections, the direction depends on the mouse's screen position. You can look into my answer to this question to get an idea of how this is done.
Now perform a collision check of the ray and your model to find the affected triangle/triangles and vertices. Calculate the Barycentric coordinates where the ray hits the affected triangle. Use them to interpolate the texture coordinates of the intersection point. From the model data, you should know, which texture is used at the target location.
Now you know where you want to paint in your models' texture. How to proceed depends on what you want to paint. In case you just want to paint a pixel, you have to overwrite the texture's pixel at this location. You can do this either on the CPU or on the GPU using the rendering pipeline.
If you don't understand some things I mentioned yet, read the links I provided and continue learning OpenGL. Things will get clearer as soon as you know the rendering pipeline and get used to working with textures. If something is still not clear after that, feel free to ask.


Why does some text on my laptop have only gray-scale pixels while other text has color pixels? In this question I generated the images below to explain why I can't use TTFs for that 1-bit display application.

But now I would like to understand from the perspective of a non-expert, why is some black-on-white text on my screen made from gray pixels, while other text is made from color pixels?

Gray on left is a screenshot from the SE question editor window, color on right is from a Finder window in MacOS.

The "Why?" should have two elements, one might be "because Apple does it that way" but the other would hopefully help me understand why Apple does it that way if that is in fact the case.

update: The answer to why does black text have orange and blue pixels names this as "sub-pixel rendering" but I still do not understand why it is in color in some places, gray in others, and if and why the color is used at all.

I've done a rough analysis of the colors used in hopes that it might teach me something, but I can't really gain any insight beyond "its a red-blue thing".

It's called subpixel rendering (there are different solutions on different platforms). Microsoft's one is called ClearType, just google for it, Wikipedia has a nice description.

Earlier, simple edge smoothing was used, antialising, that resulted in gray pixels added to the jagged lines to mask them, and although it technically created a more blurred outline, due to the way our brain works, we perceived the resulting forms more pleasing, more natural, easier to read.

Colored subpixels arrived when monitor technology moved from CRT screens to LCD where the RGB components making up a pixel are much more uniform. Very simply put, we treat our monitors as having thrice as many pixels horizontally than they actually do. When we render a character glyph, the body will be black, so no difference there. But if we calculate to need just a subpixel more to the left of a complete pixel, we will use that extra subpixel. Similarly, if we need to go just a subpixel to the right, we will use that.

But, using these individual subpixels rather than full pixels as with antialising, we will have color fringe inevitably because these subpixels are physically colored. Just look at your sample: all left edges have the same color fringe, yes, because that's the color of the last subpixel of the previous pixel. And right edges, similarly, have the color of the first subpixel of the next pixel.

What you can see in the screenshot is that the display sent to the monitor is pre-manipulated with the anticipation of the colored subpixels and displayed accordingly. Pixel colors are modified so that when they light up on the monitor screen, only those subpixels will be lit that are required by the calculations.

Monitors might be different and, more importantly, individual sensitivity to this effect is different from person to person. Apple, using only their own monitors and in line with their usual stance of "we tell you how to use it", only offers a way to switch this effect on or off in their user interface (although there is a way to select from a limited list of three different settings using the Terminal app and appropriate commands).

In Windows or Linux, where the hardware selection is much broader and people don't normally accept to be allowed only a single way, ClearType can be adjusted to your individual needs. Depending on the settings you selected, the actual color fringes will be very different.

This is just a compromise. In order to display text to be more legible, we sacrifice chrominance for luminance. Differences in intensity are more noticeable to the human brain than differences in color. So, even if it introduces color variations (which is absolutely not the intention but an inevitable consequence of the subpixels being colored), what we really see are the intensity differences. In simpler words, most of us will only perceive at that resolution and physical viewing distances that there is an extra subpixel, not that it has a different color.

And as to the final question of why gray in some places and colored in others, it's very probably a software issue. The browser you mentioned might have its own graphics rendering engine that simply works differently from what you normally have in the operating system. It's quite possible that it doesn't use the operating system's services to render text but renders everything by itself, pictures and texts as well, as a complete image and uses its own settings and techniques in the process.


What are the benefits of downscaling on modern games? I'm a software engineer and although I currently don't work on anything related to computer graphics, I understand how downscaling works, what I don't understand is why most modern games default to over 100% on my computer (GTX 1080, 1080p@144Hz monitor). For instance, Overwatch defaults to 140% render scale, which translates into a huge performance hit (stable 144 vs 60~80 fps). Why is the game doing this? I understand that downscaling makes the quality of the image a bit better, but this doesn't justify that performance hit. I read in some places that downscaling also helps to reduce some graphics artifacts, can someone please elaborate on this? I would like to know what are the benefits of downscaling and why modern games are defaulting to settings above 100%.

What you are describing is supersampling, a technique for combating aliasing (a.k.a. jaggies), which is a term from signal theory that also applies to graphics. It's a spatial domain approach. Basically it means that the sampling resolution (render target pixels in this case) makes signals with different frequencies be reconstructed as the same frequency. Or, in other words, the scene is sampled too infrequently for sub-pixel detail to be properly represented.

It is especially prevalent in modern games since the recent advances in GPU computing power have made that quality/performance tradeoff (that you deem unjustified, but unless we're talking VR, everything above 60 FPS is just buttery-smooth and undistinguishable) even possible, and because the (fairly) recently introduced physically-based shading models are much more susceptible to aliasing from sources other than just undersampled geometry: a group of artifacts collectively called shading aliasing, with specular aliasing (a.k.a. fireflies) being especially visible. Epic Games' Brian Karis explains some of that in the opening of his presentation on temporal AA in UE4 (temporal AA is just another approach to anti-aliasing, one that distributes samples in time instead of space).

To sum up: it's the brute force approach to fixing aliasing. An anti-aliasing technique that simply captures more samples of the scene, and thus captures more sub-pixel detail that would otherwise be lost.

It is also worth mentioning that while it appears superficially similar to Multi-Sampled Anti-Aliasing, it differs in the sampling pattern (SSAA simply increases the pixel grid density, whereas MSAA has a sort of sheared grid pattern) and the sampled signal resolution (SSAA is a full colour capture, MSAA cheats slightly by sampling triangle coverage more finely than colour).


Do you have to group and optimize your draw and shader calls in Vulkan? This is in reference and contrast to the discipline (or more or less requirement) of opengl to group draws that use the same materials together, to minimize making many draw calls or many switch shader calls.

So is it safe to say, that since vulkan is not state/context based, you can throw any mixture of draw commands with multiple seperate shaders/materials in the command queue, with no slowdown or overhead impact, and that even if you did the grouping optimization, you wouldn't see a speedup as well?

Vulkan will not have the CPU-time performance costs associated with state changes (validation overhead, etc). But Vulkan command buffers do have state. They have two very major pieces of state: descriptor sets and the pipeline object.

Both of these objects are monolithic, dealing with potentially lots of pieces of information. The (potentially) huge quantity of stuff these objects control should strongly suggest to you that binding new objects is not cheap. And that's in terms of GPU execution speed, not CPU validation or whatever else.

Vulkan in general tries to make slow things in the API look slow and fast things look fast. Render passes are big, huge and complicated, so clearly you shouldn't just be starting and stopping them whenever you feel like it. Pipeline objects control a huge amount of information, so binding new ones feels like a heavyweight operation.

By contrast, changing what buffers get used for vertex input feels like a short, quick, focused operation. So it's probably cheaper than changing the pipeline that uses those buffers.

So yes, you should avoid state changes in Vulkan, just as much if not moreso than in OpenGL (since many state changes in Vulkan require entirely new pipelines).


Advice on how to create GLSL 2D soft smoke/cloud shader I want to recreate this smoke/cloud effect which is used in the Arrival (2016) movie end credits.

I uploaded a sped up version of the credits to see the smoke dynamics.

http://streamable.com/n3309 49s x6 sped up video of the credits

I know how shaders work and can comfortably implement simple GLSL fragment shaders, but looking at this I can't seem to find a way to get started.

Starting points I considered:

Using a noise function: Every noise function I could find looks unnatural and unsmooth. Not what I need.
Creating 2D/3D particle simulation: This seems a bit overkill and even then I need functions that do not look calculated.
Just use some sprites and transform them randomly around: That's lame and still looks 'fake'.

If you could guide me to a starting point on how to accomplish a somewhat similar looking GLSL shader implementation that would be great!

Noise functions are definitely your friend here—FBM would be one good candidate. You’re right that it can look too uniform on its own, but if you blend multiple layers of it together, using different speeds / directions for each and maybe distorting their domains a bit, you should be able to get very close to the look of that footage. There’s lots of clever uses of noise functions on Shadertoy; for an example of something like the approach described above, check out this one.


Is it viable to create computer graphics via terrain scanning? Couldn't find any resources on this, so I'll ask here.

For example, I want to create a game model of a mountain peak.

I took lots and lots of drone footage, almost from every side, photography footage, just lots of it. (Realistically viable amounts of footage)

Could I somehow, import, or make a 3D model out of these footages so I could import it into the game.

It should be a 3D model. Thank you.

Yes that is possible using photogrammetry like @SimonF mentioned in his comment. There is plenty of commercial and free software. Peter Falkingham has a great blog with many reviews mainly for the free tools.

Here is a video of scanning a mountain using my favourite open-source solution. Specialists do much more extreme scanning using their tools, the resulting Matterhorn point-cloud dataset is used by some of my research colleagues as well.

Requirements

To cover a mountain, you will need a lot of photos. Depending on the peak size and required precision you may end up with thousands to hundreds of thousands of photos. To process all of them, it needs a lot of RAM and powerful GPU(s). I did a reconstruction for 10k photos on a Ryzen 2700X with 64 GB Ram and a 1080 Ti inside... it took 10 days to complete.

There are two critical phases:

Alignment to create the sparse point cloud is partly CPU-based and requires a lot of RAM. Once the system starts to swap to the HDD, you've lost.
Patch-match for the dense point cloud heavily uses the GPU and takes most of the reconstruction time.

Scanning procedure

In ideal case all photos would be taken in exact the same moment. When you fly around it takes time and the sun moves. Shadows change. That is often a problem at reconstruction so cloudy days are generally better. Try to get all footage on one day and as fast as possible. The atmospheric conditions may change the next day(s). I hope there is no snow and no water, those are the most difficult to reconstruct due to reflections :)


GBuffer: Framebuffer with different texture format types? I rely on this answer from Nathan Reed: How much precision do I need in my G-Buffer?

It is proposed to use two different texture formats for one Gbuffer.

However as far as I know, color texture attachments to framebuffer objects must have the same format: https://www.khronos.org/opengl/wiki/Framebuffer_Object

  Texture
      For the purposes of this article, a texture is an object that contains some number of images, as defined above. All of the images have the same format, but they do not have to have the same size (different mip-maps, for example). Textures can be accessed from Shaders via various methods.

Is it possbile to use multiple render targets with different texture formats for a framebuffer? Or should just the lowest common format be choosen and manually pack and unpack the 16-Bit values in the shader?

You have a bit of a misunderstanding there. The paragraph you quoted doesn't actually say anything about framebuffer objects yet. It just talks about a single texture. While we usually understand a texture as a single image, it is (or can be) actually comprised of a set of images. This might be because it is an Array Texture or, even more common, it just has a set of Mip Map Levels (basically increasingly smaller filtered versions of the base image, used for advanced texture filtering). And yes, within a single texture, all those images have the same format. However one such texture usually makes for one render target.

All this has nothing to do with multiple render targets yet. And you will notice that the later parts of that article that speak about the actual framebuffer attachments, specifically the part about Framebuffer Completeness don't say anything of the sort of all attachments needing to have the same format, because there is no such restriction. They don't even need to have the same dimensions. The only thing that is required is that they all have the same number of samples (if they're multisampled at all).

It is possible that an implementation still doesn't support all kinds of image formats and combinations, but there are some that are required and a "reasonable" implementation will support a wide variety of format combinations, including ones with mixed pixel bit-depths.

There was a stricter limitation on the images in the original GL_EXT_framebuffer_object extension, namely that all attachments have the same size and internal format. However, since it has found its way into OpenGL 3.0+ core (and was from there exported back as the more advanced GL_ARB_framebuffer_object extension) they loosened these restrictions.


Jitter texture generation I am currently reading the awesome paper by Jorge Jimenez about Character rendering : Next Generation Character Rendering

In the part about multi-sampled transmittance he uses some Poisson offsets and a jitter texture which contains [cos(x), sin(x)] and is used to rotate the the Poisson disk samples.

But I am wondering how do you generate such texture ?

Here is the part of the ppt which talks about this.

The texture is probably generated by picking a random angle per pixel, and populating the image with its sine and cosine, remapped into [0, 1]:

$$\theta \sim [0, 2\pi] \quad \to \quad \begin{bmatrix} \tfrac{1}{2} \cos \theta + \tfrac{1}{2} \\ \tfrac{1}{2} \sin \theta + \tfrac{1}{2} \\ 0 \end{bmatrix}$$

I did a quick test and generated something that looks pretty similar (enlarged 4x):

If you want to be extra fancy, then instead of picking independent random angles for every pixel, you might try stratified sampling or a low-discrepancy sequence. That may give better final results for whatever effect you're using the jitter texture for, due to better local sample distribution.


What are "Triangles" in 3d modeling? In 3d modeling, there is some "triangle count" which determines complexity. I am not sure but I think cubes have 12 triangles or somethong. But I see NONE. They have squares, no triangles! Can someone please help me understand what a triangle is? Where in the cube are these 12 "triangles" located?

A literal triangle, as in 3 vertices connected by 3 edges.

They are the basic building block for any other shape, any polygon can be represented with a set of triangles (called triangulation).


How to render an object that recieves shadows but does not cast them in a variance shadow mapping system? I'm currently preparing to implement variance shadow mapping based on this article. However, one point it makes concerns me:

  Rendering only casters (and not receivers) into the variance shadow map is incorrect! For the interpolation to work properly, the receiving surface must be represented in the depth distribution. If it is not, shadow penumbrae will be improperly sized and fail to fade smoothly from light to dark.

This means that the naive method of rendering an object that does not cast a shadow, but can recieve shadows from other objects - simply skip rendering it into the shadow map - will not work. How else can this be accomplished?

Not rendering a shadow caster could work but your mileage can vary. When variance is "large" a VSM is not able to exactly localize the occluders, which causes light leaks. Not rendering a large receiver (let say the floor of a room, a road, etc..) can dramatically increase the variance of the depth samples within a region of the shadow map that include the missing receiver(s) (e.g. the receiver depth is replaced by something much more distant, like the far plane).

On the other hand, if you skip rendering in the VSM some little object that is unlikely to ever be the most distant receiver in the scene, then you have some hope that things will work out (compatibly with the limitations of variance shadow maps). If you can afford the cost I suggest you to use something more robust like exponential variance shadow maps (see this work: https://software.intel.com/en-us/articles/sample-distribution-shadow-maps).


Camera Model View transformation in Ray Tracing I am implementing my own simple ray tracing and I came across some sort of confusion. When defining the verticies in the world I multiply them by the modelView transformation system (the matrix similar to the one generated by glm::lookAt function) in order to transform them to the camera coordinate. However, the verticies are invisible in the world when I do this transformation. My question is: are we supposed to transform the veriticies in Ray tracing with the model view transformation matrix as we do in OpenGL ?

The space at which you transform your vertices is completely up to you, because it depends on what algorithms and kind of effects that you are trying to achieve.

As of my personal experience, I usually shoot rays in world space because eventually we all need some sort of "world-space" acceleration data structure, such as a space-partition tree, that gathers all objects into a single space. As this data structure is probably going to be pre-computed for static objects, world space seems to be the most preferred one, as the vertices representation never change when you move the camera around.

When it comes the time to actually test the ray-object intersection, I transform the ray direction onto object space by using the inverse of the model matrix. This makes it possible for all intersection tests to be implemented as if the object were axis-aligned. Thus, you can easily use AABBs (axis-aligned bounding boxes) for coarse intersection testing, even when your models are rotated in world space. This is also a good optimisation for dynamic objects, as the AABB does not change in object space.


Back and forth rotation on Y-axis I want to write code for a cube rotating 10 degrees back and forth on its y-axis in an OpenGL application. Right now I have this written in my display method, however it depends on processor speed, and it may be more complicated than it needs to be. I would like to make the code independent of processor speed.

GLfloat rot = 10;
GLfloat angle = 0, angle2 = 0, angle3 = 0;

if (angle < rot){
   glRotated(angle, 0, 1,0);
   angle++;
}
else{
   if (angle2 < rot * 2 ){
     glRotated(rot - angle2, 0, 1, 0);
     angle2++;
    }

else{
    if (angle3 < rot ){
     glRotated(-rot + angle3, 0, 1, 0);
     angle3++;
    }

else{ angle = 0, angle2 = 0, angle3 = 0; }

In order to dissociate the rotation from the CPU speed, you need to use some sort of timing in your function. I recommend passing in a time value of some sort. The easiest thing is to pass in the number of seconds since the app started. How you do that depends on your particular OS, which you didn't specify.

Once you have a timing constant, you can generate a triangle wave to control the rotation. This formula will oscillate between 0 and 1 over time:

double rotation = 2.0 * fabs(fmod(time, 1.0) - 0.5);

You can make it go from 0 to 10 by multiplying the result by 10:

rotation *= 10.0;

Then you plug that into your glRotated() call:

glRotated(rotation, 0, 1, 0);

If you want to make the motion a little smoother, you could replace the triangle wave with a sine wave, like this:

double rotation = 0.5 * (sin(time * 2.0 * M_PI + M_PI / 2.0) + 1.0);


Energy redistribution sample re-integration I'm implementing ERPT-style energy redistribution of primary samples within a pathtracer. Specifically, to improve the sampling of defocused areas.
The core algorithm does two things:

If sample luminance is higher than threshold, I redistribute the sample energy over many backwards fired rays. This is stored in a framebuffer. The result of this is super clean!

I also store all non-redistributed samples in a separate framebuffer.

My question is about how I should recombine these two framebuffers. Which probabilities should I be using? I've tried a variety of approaches but none of them have worked perfectly. I cannot just remove the samples-to-be-redistributed and sum the redistributed samples on top. This results in discontinuities. I think I need to be using a certain weighting.
An example scenario is as follows:
Pathtraced result I start with:

Take a selection of the samples to redistribute:

Redistributing these high intensity samples in separate framebuffer:

How can I re-combine the redistributed & unredistributed results?
Any help would be appreciated!

It sounds like you yet have to adjust the importance function of the Markov Chain according to the way you assign paths to each of the two estimators, path tracing and ERPT.
If you think about it in the framework of MIS, you have two estimators with binary weights (according to your threshold). Now, the simplest (but inefficient) option to combine them would be to always run ERPT and path tracing for all paths, applying the weight to the splats only (this weighting is what I understand you do by separating them into different framebuffers).
In your case, however, you only enter the ERPT chain for the paths that you want to further explore, therefore you do not actually enter the chain in a stationary distribution proportional to the measurement contribution. This breaks ERPT, as the startup bias removal no longer works correctly. To fix it, you need to additionally account for your path selection strategy in the importance function such that it matches the initial distribution of paths entering ERPT: The MIS weight of the ERPT estimator needs to become a part of its targeted distribution, preventing energy from running off into paths that would never actually trigger an ERPT run. Thus, you should be able to make the startup bias removal work with your path selection strategy.


What is a "Vulkan Extension"? When creating a VkInstance, you need to setup the API's extensions that are going to be used by your application (for example you need an extension to interface with the window system on desktop platforms).

What are these extensions? What do they represent? Are they just a way to tell Vulkan to use a certain feature implemented by a driver?

Are they just a way to tell Vulkan to use a certain feature implemented by a driver?

Well... yes. Vulkan extensions are simply additional features that Vulkan implementations may provide if they so choose to. They add new functions, structs, and valid enumerators to the API, and they can change some of the behavior of existing functions.

You have to detect whether the implementation supports these extensions before creating an instance/device. And you have to explicitly ask for such extensions when you create the instance/device.


Smart half edge iteration? In my HE implementation, half edges are stored in an array. When I iterate over the edges, I color all the HE black, and when I do an operation on an edge (e.g edge splitting) I mark both the current half edge and its pair blue. And in the loop I skip blue edges.
This essentially stops me from applying the same operation to an edge twice.
This however requires O(n) additional memory and iterating over every edge twice (once for each of the HE that compose the edge.)
    for(uint i=0; i < edge_num; i++)
    {
        if(mesh.edges[i].color != BLUE)
        {
            /*Do stuff*/
           mesh.edges[i].color = BLUE;
           mesh.edges[i].pair.color = BLUE;
        }
    }

I am wondering if there is a smarter way. In particular, a way that avoids branching. I know I could do something like depth first search or breath first search, but that's likely to be slower than just skipping over edges.

One possibility would be to create a bit array with 1 bit per half-edge. When you start iterating, initialize them all to 1, then clear the bits of each half-edge and its partner as you iterate.
The iteration can be done using __builtin_clz (GCC, clang)  or _BitScanReverse (MSVC)  [edit: or std::countl_zero in C++20!] to efficiently extract the next 1-bit without branching on every bit. Although you do then need a nested pair of loops, where you loop over the fields in your bit array, and then loop extracting bits until the current field is zero.


Point respect to plane How do I classify a point with respect to a plane, returning whether the point is inside, outside, or coplanar?

I know that if a point is on the plane the Dot product + the plane distance would equal to 0, but what does inside, outside, and coplanar mean. Also how it is computed.

Vec3 v = myPoint - pointOnPlane;
float vDotPlaneNormal = dot(v, planeNormal)
if (vDotPlaneNormal == 0)
    printf("point on plane");
else if (vDotPlaneNormal > 0)
    printf("point above somehow...");
else
    printf("point under sadly...");

If you wonder what is pointOnPlane. In fact it can be any point that lies on the plane. Imagine this as a sheet of paper; draw two lines on this sheet to define the center of the piece of paper, imagine that this point is aligned with the world coordinate system origin and that sheet is in xz plane of that coordinate system (in which y defines the up vector). So the paper is horizontal in a way. Now move the sheet around. The pointOnPlane can be the coordinates of the point you drew in the center of the sheet of paper with respect to the world origin.
All you have to do is rotate that sheet of paper. That rotation is given to you by the plane's normal.
The rest is basic trig. If the point P is one the plane, the dot product between the vector defined by (P - PointOnPlane) and the plane's normal will be equal to 0. The dot product tells you whether two vectors are orthogonal, on the same side of the plane or point in opposite directions.
FASTER SOLUTION - EDIT
As mentioned by SimonF in a comment, you don't really need the pointOnPlane parameter and save the subtraction between this point and the tested point:

As you imply, a plane can be represented by its normal, $N$, and a scalar, $d$.
A point X is on the plane if $N \cdot X+d = 0$. As for inside and
outside typically you define a rule, e.g., that the normal points
"out" from the plane which just means that if $N \cdot X + d > 0$, then
$X$ is above the plane, while $N \cdot X + d < 0$, implies it's below.

You can test/use both techniques.


Pixel density / color channel depth ratio for dithered gradients Dithering increases percieved color depth by adding a "noise" to an image on transform from high color depth (e.g. 32-bit floating point) to lower color depth (e.g. 8-bit integer). A prerequisite is pixel density being high enough for the brain to "blend" multiple pixels together. It is also dependent on viewing distance. I´m most interested in phone or desktop workstation distance. Dithering algorithm also matters but for the sake of the question lets assume it is "perfect".

RGB8 (TrueColor, 16 million colors) requires dithering for nice smooth gradients and tend to get really nice for your average monitor. RGB5 (HighColor, ~32k colors) requires dithering for gradients to look nearly smooth, but I think pixel density start to really matter at this point. I´m not sure if RGB4 (~4k colors) or R3G3B2 (256 colors) can ever have smooth gradients but given high enough pixel density it should be possible.

Is there any rule of thumb when it comes to pixel density / color channel depth ratio for a dithered image to appear having smooth gradients as percieved by the brain?

There has been quite some research into this using Barten contrast sensitivity function. It is the current formula behind the Dolby Perceptual Quantizer as featured in SMPTE 2084 and HDR10.

This, coupled with a colour appearance model such as the work behind Dr. Mark Fairchild's CIECAM02, can result in very accurate predictions of quantization depth.


maximal convex patching in Computer graphics Given a 3D object in Computer graphics, whose surface is represented as a 3D triangular mesh (mesh of 3D triangle objects), I need to find the maximum continual Convex patches on the surface of the given 3D object.

I am using OpenGl to render the graphics within a C++ program. What kind of methods or algorithms should I use to find the convex patches.

I have to apply different colors to the different convex patches on the object to signify the selection.

Say I have a sphere then the whole sphere is one maximal convex patch. Any portion of the sphere surface will be a convex patch, by maximal I mean the maximum continuous convex patches that can be found. Well in the rendering, depending on the viewing angles, the maximal convex patches visible to the viewer will have to colored. I have to report all the convex patches, each patch being the maximal in that area.

I am using "maximal" to mean a convex surface which is not a subset of a larger convex surface, rather than to mean the largest convex surface that exists in the triangle mesh. The patches need to be strictly convex (never flat).

Start from any triangle. Traverse it's edge's and check that the angle between the two triangles is less than 180deg. If it is add it to the current selection and continue expanding.

The check is actually really simple if you use vector geometry. Say A - B is the common edge with C on the selected side and D on the other. Then just check if dot((D-B), cross((A-B), (C-B)) < 0.

Single triangle patches should be ruled out, as for us to determine convexity, it has to span across several triangles.
We have to keep sampling until we've tested everything.  When a convex patch continues further, it's size keeps on increasing by including the neighbouring triangles that satisfy convexity, until the patch reaches the maximum size it can attain.


Which perspective projection matrix to use I've seen two different perspective projection matrices being used but I want to know which one of them is the best. I think that they both output the same but I'm not sure. So if anyone can help me select which one to use I would greatly appreciate it

Even though you can find multiple slightly different formulations for the perspective matrix on the internet, they all do more or less the same thing. They project everything inside a space with the shape of a truncated pyramid (your view frustum) into a space with the shape of a quad and a defined value range (clip space).
Most differences are related to how you define the shape of your frustum. For example, your first matrix uses a height, a width, and an angle together with z-far and z-near value. The second one uses just an x- and y-FOV and the z-far and z-near values to describe the shape of the frustum. This explains the differences in the upper left section.
So what about the lower right section? One big issue here is, that you can choose to use column or row vectors for your vertices/points. Column vectors are multiplied on the right-hand side and row vectors on the left-hand side. The choice you make determines if you have to transpose (switch columns and rows) your matrix or not. The matrices you presented are just targetting different vector representations. I can't tell you without testing/rereading which one is meant for which representation, but if you transpose one of them, they start looking quite similar.
There are still some other minor differences like varying signs, different factors, or a missing addition.
The derivation of your second matrix can be found here.
I can't tell you exactly where those small differences in the first matrix come from, but you can find some possible explanations in this link (Have a look into the "Are There Different Ways of Building this Matrix?" section).
Another good read is this link here.
Which matrix is "correct" depends on how your Graphic API defines its coordinate systems and it's clip space (For OpenGL - you can take the one from the last link I provided). However, there is no "best" choice because they all do more or less the same. Unless you have no special needs for your program (depth precision) or other fancy ideas I can't think of now, just pick the one suited for your API with the inputs (FOV vs width+height+angle) that you like most and start having fun. ;)


sdl2: Compiler throws "Undefined Reference" I'm trying to learn computer graphics, but when I tried to get this very simple windower running with c

#include <stdio.h>
#include <stdlib.h>

#include <SDL2/SDL.h>

static const int width = 320;
static const int height = 180;

int main(int argc, char **argv)
{
    SDL_Init(SDL_INIT_VIDEO);

    SDL_Window *window = SDL_CreateWindow("Hello, SDL2", SDL_WINDOWPOS_UNDEFINED, SDL_WINDOWPOS_UNDEFINED, width, height, SDL_WINDOW_OPENGL);

    SDL_DestroyWindow(window);
    SDL_Quit();

    return 0;
}

I try to compile and...

vke@komputilo:~/dev/MiaLudo/source$ gcc -g main.c -o ../ludo
/tmp/ccfVU5ua.o: In function `main':
/home/vke/dev/MiaLudo/source/main.c:11: undefined reference to `SDL_Init'
/home/vke/dev/MiaLudo/source/main.c:13: undefined reference to `SDL_CreateWindow'
/home/vke/dev/MiaLudo/source/main.c:15: undefined reference to `SDL_DestroyWindow'
/home/vke/dev/MiaLudo/source/main.c:16: undefined reference to `SDL_Quit'
collect2: error: ld returned 1 exit status

Hoping someone can help me out, I'm running Trisquel Linux if that's important

The problem is definitely a lack of linker flags.
It can see that you want to include <SDL/SDL2.h>, which is a sort of promise that said functions contained in that header have been built and will be available; but when your linker looks to find the actual machine code, it needs a little more than the header. It needs the library itself.
For SDL2, you would usually use something like:
gcc -g main.c -o ../ludo -lSDL2

On Linux, this is more-or-less the same command, but for complete definitions it also instructs the compiler to look inside /usr/lib/x86_64-linux-gnu/libSDL2.a (or wherever it is on your system). Note that the lib prefix on the file name is assumed, and the extension is irrelevant.
Do that, and your program should compile. Note that some SDL extensions, like Image and Mixer, require their own libraries. An SDL-specific shortcut to ensure that you've got everything is to use the handy pkg-config, like so:
gcc -g main.c -o ../ludo $(pkg-config --cflags --libs sdl2)

Which will be interpreted by the terminal as
gcc -g main.c -o ../ludo -D_REENTRANT -I/usr/include/SDL2 -lSDL2

This is less generic however, a bit more paranoid, and a bit of a cheat; but perhaps it will help you.
The thing to remember with C is that a function needs to be declared to be used in code, and that's compiler-end; but to be built into an actual stand-alone program it also has to be defined, which is where the post-compiler linker comes in.


Building a lookat camera matrix I was reading this article and found something that caught my attention.
They say they are using column-major for their matrices so, in the code they have under the section Look At Camera they construct the orientation and translation.

$
R=\begin{bmatrix}r_x & u_x & f_x & 0 \\
r_y & u_y & f_y & 0 \\
r_z & u_z & f_z & 0 \\
0 & 0 & 0 & 1\end{bmatrix}
$

$
T=\begin{bmatrix}0 & 0 & 0 & 0 \\
0& 0& 0& 0 \\
0& 0& 0& 0 \\
-e_x& -e_y & -e_z & 1\end{bmatrix}
$

$
RT_1 = \begin{bmatrix}r_x & u_x & f_x & 0 \\
r_y & u_y & f_y & 0 \\
r_z & u_z & f_z & 0 \\
0 & 0 & 0 & 1\end{bmatrix}\begin{bmatrix}0 & 0 & 0 & 0 \\
0& 0& 0& 0 \\
0& 0& 0& 0 \\
-e_x& -e_y & -e_z & 1\end{bmatrix}=\begin{bmatrix}r_x & u_x & f_x & 0 \\
r_y & u_y & f_y & 0 \\
r_z & u_z & f_z & 0 \\
-e_x& -e_y & -e_z & 1\end{bmatrix}
$

They say this could be optimized if instead of doing the matrix multiplication we set the last row to be the dot product between camera position and the $\vec{r}, \vec{u} $ and $\vec{f}$ vectors.

$
RT_2 = \begin{bmatrix}r_x & u_x & f_x & 0 \\
r_y & u_y & f_y & 0 \\
r_z & u_z & f_z & 0 \\
-<\vec{r}, \vec{e}>& -<\vec{u}, \vec{e}> & -<\vec{f}, \vec{e}> & 1\end{bmatrix}
$

How is this possible? I don't see how $RT_1 = RT_2$ could give the same matrix. What am I missing? The only way I can see this is true is inverting the order of multiplication, i.e. $TR$

Because all matrices are column-major,
the translation matrix $\mathbf{T}$ should be

$$
\mathbf{T}=\begin{bmatrix}
1 & 0 & 0 & e_x \\
0 & 1 & 0 & e_y \\
0 & 0 & 1 & e_z \\
0 & 0 & 0 & 1
\end{bmatrix}
$$

I thought you mistakenly treat the eye as a row vector just because they were written as a single line in the source code

mat4 translation = {
    vec4(   1,      0,      0,   0 ),
    vec4(   0,      1,      0,   0 ),
    vec4(   0,      0,      1,   0 ),
    vec4(-eye.x, -eye.y, -eye.z, 1 )
};

Another issue in that article is that the variable names are misleading:
the orientation denotes $\mathbf{R}^{-1}$, and the translation denotes $\mathbf{T}^{-1}$(I think it is not a good practice even they commented on these variables).

The final view matrix is:

$$
\begin{eqnarray*}
\mathbf{M} & = & (\mathbf{T}\mathbf{R})^{-1}\\
& = & \mathbf{R}^{-1}\mathbf{T}^{-1}\\
& = & \mathbf{R}^{T}\mathbf{T}^{-1}\\
& = &
\begin{bmatrix}
r_x & r_y & r_z & -<\vec{r}, \vec{e}> \\
u_x & u_y & u_z & -<\vec{u}, \vec{e}> \\
f_x & f_y & f_z & -<\vec{f}, \vec{e}> \\
0 & 0 & 0 & 1
\end{bmatrix}
\end{eqnarray*}
$$

and the implementation in that article is correct.


How are colors really mixed? I'm kind of stuck on this one. When following a conventional, high-level shading tutorial, you'll come across these images that are like ambient + diffuse + specular = result, but I can tell that it's not straight addition (otherwise the picture would seem overexposed).

What's the "formula" for mixing two rgb colors?

With an old-style Phong shader (as in your example), they really are just added together. Each of those things is a contribution to the overall amount of light, so they're just added together, the same way you add together the contribution from each light source.

Your intuition is right that you often end up with overexposed specular highlights with this style of shading. In a modern, physically-based pipeline, it's common to instead use an energy-conserving shader. The contributions of each layer (specular, clear-coat/lacquer, diffuse, subsurface) are still added together, but the amount of light in each layer reduces the input amount of light for the next layer. That is, if the surface is giving a strong specular reflection, it's reflecting most of the light, so less light is available for the diffuse layer. In the same way, any light that's reflected by the diffuse layer reduces the amount of light available for subsurface scattering.


Link between blue noise sampling and blue noise textures? You can have blue noise sampling like these poisson disc samples:

And you can have a blue noise texture like this:

I get that in the first image, there is one input (the index of the sample) and two outputs (the x,y coordinate of the point) and that the second image is basically the reverse where there are two inputs (the x,y coordinate of the sample) and one output (the value of the point).

I'm curious though, how are these related?

If you take the DFT of the second image, you can see that it has more high frequency components than low, but I'm not sure how you'd take the DFT of the first set of data points.

I'm wondering if it's possible to take other low discrepancy sequences (say, halton, or jittered grid) and make a texture out of the idea, like the second image?

The missing link between sample locations and the greyscale noise texture is "ordered dithering".

Ordered dithering is a list of pixel locations with a "rank" (order) for each pixel. If you have a white background and want to add two black dots, you add them at the locations for the two pixels rank 0 and rank 1.

Choosing how to rank the order of the pixels to turn on can vary dramatically with different results though. For instance, a bayer matrix is a specific ordering of the points, and blue noise sample points are as well. White noise is just shuffling the points so that they have a random ordering.

How we get from this "ordered dithering" (stippling) to the greyscale color noise images is that we divide each point's rank by the number of points to get a value from 0 to 1, and use that as the points greyscale color.

That gives you the greyscale noise textures.

The greyscale blue noise texture was created with the "void and cluster" algorithm which makes it so that each new dot placed goes in the middle of the largest void.  This has the nice property that you can threshold the blue noise texture at any value, and the result will be blue noise samples of the desired density.

This paper is a great read that talks about these things more deeply:
http://cv.ulichney.com/papers/1993-void-cluster.pdf


How could I use bent normal map This might be a general question but I have some difficulties about understanding baked bent normal maps and how to use it inside a shader in Unity.

From what I understand, bent normals are especially used with occlusion but I don't really get what they contain and how to use them.
The other strange thing is that when baking them with different tools, it gives me different output, so it's hard for me to interpret them.

Here are some examples:

This is made from Substance Designer in World Space. Seems really close to a simple normal map in World Space.

This is made from Substance Designer in Tangent Space. It doesn't seem to contain really relevant informations as the one above.

This is made from Maya Turtle. It seems to contain interesting information but... what space is that ?!

This is the same as the one above but converted to Unity using Handplane. What can I do with this...

It seems that Unity is using tangent space normal maps but I don't have any control on the import process and I don't know what they are doing. Regular normals are then unpacked in converted into world space normals using TBN matrix.

The fact is that at this point, any clue or advice should be really useful.
I don't know if my baked maps are right or usable and I don't know how to use them inside my shader.

Here is an interesting answer from Styves on gamedev.net.

  Calculating bent normals is just an extension of AO calculation where
  the direction of each sample is also averaged along with the occlusion
  amount.

  The bent normals from substance painter seem correct. In any region
  that there is no AO, your normal won't be bent in any direction,
  therefore in tangent space it's pointing straight up along the surface
  normal. This is why you only see details in the ears.

  I have no idea how Unity handles the bent normals but the usual way is
  to, instead of applying AO as a standard multiplier onto the lighting
  result (which gives this weird muddy look), use the dot product of the
  bent normal and the light instead. This is how we handle SSDO in
  CryENGINE for example.

  I've only ever used a texture-based version of this technique once,
  however in this version the bent normals also contained information
  from the original tangent space normal map, and was used directly
  during lighting in place of the original.


Using original vertices coordinates after clipping Clipping is down in homogeneous clip space, therefore after multiplying the vertices by the model-view matrix. But as we do clipping in clip space, the position of the new vertices that we interpolate is in clip space, so how can I translate back those new vertices coordinates into view space?
I could perform the inverse transformation, but it seems costly. So how exactly do clipping work in the graphics pipeline? I mean, how can we get the position of the new generated vertices in view space?
For example, with OpenGL, I would do the following in the vertex shader:
layout (location = 0) in vec3 pos; // vertex position in model space

out vec3 posViewSpace;

// M V P are the model view projection matrices

void main() {
    posViewSpace = V * M * pos; // I want the pos in view space for shading
    gl_Position = vec4(P * V * M * pos, 1.0); // Give OpenGL the position in clip space
}

But how can this work, because posViewSpace that will be given to the pixel shader has nothing to do with the new vertices generated after clipping? Or maybe it doesn't matter, because posViewSpace will be interpolated from the original vertices, but the pixel shader won't notice? Will the interpolation be correct and usable in the pixel shader, even if it's performed from the original vertices and not the new generated vertices?

A vertex consists of not just the position, but all of the values passed from the vertex processing stage to the rasterizer. So if a vertex gets clipped, the new vertices generated in that process must generate new values for those vertex processing outputs. It does this using the same math that it uses to generate the new vertex position.


Z-buffering implementation with small triangles For research purpose, I have to implement z-buffering algorithm to render 3D face models. But I got a problem that the triangles are too small to be rendered (i.e. there is no pixel covered completely  by any of the triangles).

I've tried to increase the resolution of the output image (so that the triangles get bigger) but it's been $10,000 \times 10,000$ so far (I think it's pretty high) and my computer can't suffer from higher resolutions.

Can you give me some solution for this problem ?

Assuming the small triangles form continuous surfaces, although many of the small triangles "fall through the gaps" (so to speak), some should still cover the samples. If on the other hand, they are separate/independent triangles then it will alias very badly.

One possible alternative to just doing higher resolutions would be to simulate an Accumulation Buffer. In essence, do N XxY pixel renders, but for every render, offset the X&Y screen coordinates by a sub-pixel amounts.

You can then combine the N images, (perhaps initially with a simple box filter, but a Gaussian would be better) to produce your final XxY image.

  Can you explain what does "offset the X&Y screen coordinates by a sub-pixel amounts." mean ?

Perhaps it'd be a good idea to read  Haeberli and Akeley's original paper but I'll try to put in a bit of an explanation on how I would try to do the XY shifts. (I apologise in advance if I've made some fundamental flaw in the following logic...)

Let's assume we want to shift the image by $(x_{offset}, y_{offset})$ in pixels where, probably, $|x_{offset}| < 0.5$ and $|y_{offset}|<0.5$.

We want to add this offset to each vertex after it's been projected into screen coordinates. Now since we don't have much control of the pipeline post clipping and projection, we'll need to do that by adjusting the projection matrix. If we assume our default projection matrix, P, does something of the following:
$$(X_{world}, Y_{world}, Z_{world}, 1).P=(X_{clip},Y_{clip},Z_{clip}, W_{clip})$$
i.e. maps from world space to clip space, then to shift the points in image space we'll need an additional "matrix" adjustment which, off the top of my head, will be something like...
$$Adjust= \begin{bmatrix}
    1 & 0 & 0 & 0 \\
    0 & 1 & 0& 0 \\
    0 & 0 &1& 0 \\
    scaledX_{offset} & scaledY_{offset} &0& 1 \\
    \end{bmatrix}
$$
The "scaling" should take into account the resolution of the target image.

Create a new, per offset, projection matrix from $P.Adjust$ for each "offset" image you want to accumulate.


get partial view points from 3d surface mesh I have a 3d surface mesh.

Say the camera is located in some points and I want to get the visible points from the 3d surface mesh from the camera view.

I believe this kind of process would be done before 3d rendering in graphics.

I want to know how it generally works and fast, simple algorithm for the problem. It would be better, if there is some matlab function library to do this task.

Note that 3d rendering is not my purpose... I just want to get the visible point set from the 3d surface mesh.

In your case you would test every point to see if it lies inside your camera frustum, this is basically a set of point-vs-plane tests. This is not a fast way to do do things but the simplest to understand and implement.


How can I implement this BSDF? I am trying to implement a path tracer, but I have some problem. The class Spectrum is color. I implemented a mirror material like this:

Spectrum MirrorBSDF::f(const Vector3D& wo, const Vector3D& wi)
{
    return reflectance * (1.0 / abs_cos_theta(wo));
}

The reflectance is a Spectrum variable. And now I want to add a material "Glass" that can reflect and refract. It meets the Fresnel theorem. Can someone teach me how to implement this? The function f is used like this:

Spectrum light_L = light->sample_L(hit_p, &dir_to_light, &dist_to_light, &pdf);

// convert direction into coordinate space of the surface, where surface normal is [0 0 1]
Vector3D w_in = w2o * dir_to_light;
double cos_theta = std::max(0.0, w_in[2]);
Spectrum f = isect.bsdf->f(w_out, w_in);

// TODO: Construct a shadow ray and compute whether the intersected surface is
// in shadow and accumulate reflected radiance
Ray shadow_ray(hit_p + EPS_D * dir_to_light, dir_to_light,
               dist_to_light - (EPS_D * dir_to_light).norm(), 0);
if (!bvh->intersect(shadow_ray))
{
    L_out += (f * light_L * (cos_theta * scale / pdf));
}

I implemented it.But something is error.Where is error?

When implementing transmission into a path tracer you have to keep in mind that neither BRDF's nor your previous sampling method are gonna cut it. You will need to think about what happens to light on both sides of your surface, which is where BTDF's and, more generally, BSDF's come in (the T and the S stand for Transmittance and Scattering, respectively).

It can be somewhat non-trivial to implement transmission to a path tracer that currently only supports reflection.

While i can't offer help with implementation details i can give you a few starting points and things you might want to read while you embarc on implementing this capabilty on your path tracer.

On refraction.

Pure and/or glossy refraction is a very common way of transmission, it is the type of transmission that happens on glass and probably what you are looking to implement.

Snell's Law.

Light directions in refraction follow a simple mathematical law.

$$n_1\sin(\theta_1)=n_2\sin(\theta_2)$$

this reads as: the sine of the transmitted angle $\theta_2$ equals the sine of the incoming direction $\theta_1$ times the ratio of the "outside" index of refraction and the "inside" index of refraction $\frac{n_1}{n_2}$

The GGX paper.

There is a wonderful paper titled "Microfacet Models for Refraction through Rough Surfaces" that goes over a good piece of the theory behind microfacet models for refraction and presents a really quite good model for it.

Fresnel and Monte-Carlo.

when rendering a material that has both reflective and transmissive components there are a few different approaches that you can take to mix the two.

two main ways come to mind

1. The Monte-Carlo way

Do a "coin flip" with probabilities weighted according to the calculated fresnel value for each rendered point. Depending on the result you do transmission or reflection

2. The recursive branching way

Generate a reflected ray and path trace.
Generate a transmitted ray and path trace.
Mix the two found color values according to Fresnel value for every rendered point.

I really hope this information is of use in solving your issue, and good luck! Great to see new users on the site.