Ask for detailed derivation of a formula in "Advanced Global Illumination" Attached is page 34 of "Advanced Global Illumination", 2nd edition, by Dutre et al. I don't understand how (2.22) is derived from (2.21) and the given incident radiance distribution $L(x\leftarrow\Psi)=L_{in}\delta(\Psi-\Theta)$. Could you please give me a detailed derivation? Thanks a lot!

The only thing you really need to know in order to derive (2.22) from (2.21) is that the $\delta$-distribution satisfies

$$ \int \delta(x) f(x)\,\text{d}x = f(0). $$

(The $\delta$-distribution is not a function in the common sense, so the above integral is merely a useful notation and not to be read as e.g. a Lebesgue or Riemann integral.)

Unfortunately, the notation that the authors use for the substitution of the incident radiance distribution is misleading. The $\Theta$ in $L(x\leftarrow\Psi) = L_{in}\delta(\Psi-\Theta)$ is completely unrelated to the integration variable $\omega_\Theta$, so I chose to denote it by $\tilde\Psi$ instead.

Here is the derivation of (2.22) from (2.21):

Substitute $L(x\leftarrow\Psi) = L_{in}\delta(\Psi-\tilde\Psi)$:

$$\frac{\int_{\Omega_x} \int_{\Omega_x} f_r(x, \Psi \rightarrow \Theta) L_{in}\delta(\Psi-\tilde\Psi)\cos(N_x,\Theta)\cos(N_x,\Psi)\,\text{d}\omega_\Psi\,\text{d}\omega_\Theta}{\int_{\Omega_x}L_{in}\delta(\Psi-\tilde\Psi)\cos(N_x,\Psi)\,\text{d}\omega_\Psi}$$

Substitute $\Psi$ with $\Psi+\tilde\Psi$:

$$\frac{\int_{\Omega_x} \int_{\Omega_x-\tilde\Psi} f_r(x, \Psi+\tilde\Psi \rightarrow \Theta) L_{in}\delta(\Psi)\cos(N_x,\Theta)\cos(N_x,\Psi+\tilde\Psi)\,\text{d}\omega_\Psi\,\text{d}\omega_\Theta}{\int_{\Omega_x-\tilde\Psi}L_{in}\delta(\Psi)\cos(N_x,\Psi+\tilde\Psi)\,\text{d}\omega_\Psi}$$

Use the above property of the $\delta$-distribution:

$$\frac{\int_{\Omega_x} f_r(x, \tilde\Psi \rightarrow \Theta) L_{in}\cos(N_x,\Theta)\cos(N_x,\tilde\Psi)\,\text{d}\omega_\Theta}{L_{in}\cos(N_x,\tilde\Psi)}$$

Simplify:

$$\int_{\Omega_x} f_r(x, \tilde\Psi \rightarrow \Theta) \cos(N_x,\Theta)\,\text{d}\omega_\Theta$$

Now you only need to notice that we want equation (2.21) to be true for all such "test functions" $L_{in}\delta(\Psi-\tilde\Psi)$ for the incident radiance, i.e. for all possible values of $\tilde\Psi$.


Particle & Fluid Simulation: Where to learn? Where can I learn about particle and fluid simulation? Is there a good book available?  I want to couple volume rendering with fluid/particle simulation to be able to finally render smoke and ocean with foam etc. Is there a chapter in PBRT on it?

PBRT handles only the rendering part, not the simulation part. There are several books out on game physics that handle fluid simulation, the one on my bookshelf is David Eberly's "Game Physics".


Cost of parameter passing in webgl pixel shaders? In a webgl pixel shader, all functions are inlined as i understand it, however you can have parameters that are marked as in versus being inout meaning that their value can change but the value won't persist outside of the function call.

Does this mean that the shader must make a copy of the value for the function to work with when it is an in value?

Are shader compilers/optimizers smart enough to know when they don't need to make a copy, or is it best to really just mark up all parameters as inout and make sure and not modify the ones you don't want modified, if performance is the primary concern?

Thanks!

My experience working with shader compiler stacks a few years back is that they are extremely aggressive, and I doubt you will see any perf difference, but I would suggest testing as much as you can.

I would generally recommend providing the compiler (and human readers) with more information where the language allows it, marking parameters according to their usage. Treating an in-only parameter as in/out is more error prone for humans than compilers.

Some detail: shaders run almost entirely using registers for variables (I worked with architectures that supported up to 256 32 bit registers) - spilling is hugely expensive. Physical registers are shared between shader invocations - think of this as threads in a HyperThread sense sharing a register pool - and if the shader can be complied to use fewer registers there's greater parallelism. The result is shader compilers work very hard to minimize the number of registers used - without spilling, of course. Thus inlining is common since it's desirable to optimize register allocation across the whole shader anyway.


How to scale and offset a cubemap using sphere mapping I am trying to scale and repeat a Cubemap with Latitude-Longitude mapping layout just as you would do with classical UV mapping but without any interesting result. This should be used as a skybox.

This comes from the fact that the coordinates are in 3D space and we can't apply this simple formula texcoord.xy * scale.xy + offset.xy

How would you handle such features : scaling which involves tiling and offsetting.

samplerCUBE _TexCube;
texCUBE(_TexCube, float3);

This doesn't really answer the question but is a workaround. Beware it's hack time :)

Instead of using the texCUBE and samplerCube I'll be using the tex2D method and a simple sampler to sample the texture.

As I am using a sphere mesh, I generate 2D UVs like this.

float x = atan2(vertex.x, vertex.z) / PI;
x = x / 2.0 + 0.5; // To remap from 0 to 1
float y = vertex.y / 2.0 + 0.5; // Also remap from 0 to 1
float3 color = tex2D(sampler, float2(x, y) * scale.xy + offset.xy);

It's a bit hacky but using this I have 2D texture coordinates for my texture.


Experimental real-time shadowing techniques? The three most popular shadowing techniques for real time applications are:

Shadow maps

Advantages:

Fast
"Simple"

Disadvantages:

Numerical limitations lead to artifacts and jaggy shadows
You only get the information of the first object hit by the light, not the last (important for more photorealistic effects)

Volume shadows

Advantages:

Almost no numerical instability
Hardware built-in support through stenciling

Disadvantages:

Slower than shadow maps
Finding the last object that the light hits isn't easy

Volumetric rendering

Advantages:

Analytic precision
Allows for great photorealistic shadows
Finding the last object hit by the light is trivial

Disadvantages:

SLOW, VERY SLOW
Has an awful asymptotic running time for ray intersection calculations (linear in the number of primitives without hashing)

Are there any new experimental shadowing techniques that can improve on the advantages of any of these while eliminating some of the disadvantages?

There’s an interesting technique that’s been used in demoscene and Shadertoy projects for a while, and an analogue of which recently made it into Unreal Engine 4: using signed-distance fields and raymarching to produce high-quality soft shadows with accurate penumbras.

There’s a good writeup of the idea, with some visual examples, by Íñigo Quílez here—I’m not sure whether he invented the technique or was just one of the first to write about it:

  The trick is to think what happens when a shadow ray doesn't hit any object, but was just pretty close to do so. Then, perhaps, you want to put that point you are shading under penumbra. Probably, the closest your point was to hit an object, the darker you want to make it. Also, the closest this happened from the point you are shading, the darker too. Well, it happens that as we raymarched our shadow ray, both these distances were available to us! … So, we can simply compute a penumbra factor for every step point in our marching process and take the darkest of all penumbras.

The documentation on the corresponding Unreal feature is here; their description of the technique is a little vaguer, but still helpful.

  To calculate shadowing, a ray is traced from the point being shaded through the scene's Signed Distance Fields (SDF) toward each light. The closest distance to an occluding object is used to approximate a cone trace for about the same cost as the ray trace. It allows for high-quality area shadowing from spherical light source shapes.

The main advantage of this approach is its high visual quality—most other techniques don’t handle varying penumbras well, and it doesn’t suffer as much from the aliasing problems of shadow maps. The main disadvantage is needing a distance-field representation of your scene geometry to trace rays through; unless you built it that way in the first place (there’s a fascinating talk about how the team behind “Dreams” did exactly that), you’ll need a way to construct the SDFs out of your mesh geometry, which is an expensive and sometimes error-prone operation.

UE4 handles the SDF generation by building them offline beforehand and then stamping them into a volume around the camera at runtime (more information on that here), but it’s not a cheap process so UE’s approach is only viable on high-end hardware for now. That said, if you do already have your scene geometry in SDF form, it’s quite easy to implement—see this Shadertoy for an example.


Importance sampling a cosine distribution Let's take a look at this blog article - https://schuttejoe.github.io/post/ggximportancesamplingpart1/ and image presented as a result of implementing it:

Inside above link, you can see description for this image saying

In order to have something to compare to on the left I generated an image by importance sampling a cosine distribution and on the right I generated an image using the technique described above. [...]

Based on the fact that image on the right was derived based on GGX pdf, how was image on the left generated? I assume we have our standard equation but what does it mean that it's generated with importance sampling a cosine distribution?
$$
f(w_i,w_o) = \frac{F(w_i, w_m) ~ G_2(w_i,wo_,w_m) D(w_m)}{4 ~ |w_i \cdot w_g| ~ |w_o \cdot w_g|} * PDF[?]
$$

You can sample using absolutely any distribution you want, as long as you weight the results by dividing by the pdf of the sampled distribution. It will converge to the right answer (as long as the distribution is nonzero everywhere that you want to integrate). Different distributions will give different amounts of variance though. The trick with importance sampling is to find a distribution that minimizes variance while still being cheap to compute. A distribution similar to the shape of the function you're integrating, or equal to some factor of that function (so that it cancels when you divide), works best. In other words you try to guide the sampling toward areas that are more important to the result, hence, "importance" sampling.
So what the author is saying is the image on the left was sampled with $\omega_o$ (or maybe $\omega_m$, I'm not sure) drawn from a cosine hemisphere distribution, just as an example of a distribution that does not match the BRDF very well. Then they calculate the samples as $f(\omega_i, \omega_o) / p_{\cos}(\omega_o)$ where $p_{\cos}$ is the pdf for the cosine distribution. This is just to show you that it gives the "correct" result, but with a ton of variance. (Although it looks like there are some black areas on the left image that suggest it's not really converging to the correct answer...but let's assume that's a bug.)
Then the image on the right is sampled with (presumably) $\omega_m$ drawn from a distribution matching the $D(\omega_m)$ term from the BRDF. Then when you calculate the BRDF you can leave out the $D$ factor since it's canceled by the $D$ factor in the pdf. (The pdf probably still has some normalization and some conversion of probability from $\omega_m$ to $\omega_o$, so you still have to divide by those factors.)


In place sorting of a half edge DS? Cross posting from SO because I didn;t know where to put this question.
I have an implementation of the half edge and I am trying to sort the edges such that edge n+1 is the pair of edge n.
Setup
I want to do this in place to save memory and I discovered a theoretical wall I am not sure how to solve.
Let's assume the HE are stored in a list and that you can access their members through self defining getters like .Pair() or .Next().
The idea of the algorithm is very simple.
for(uint i=0; i<half_edges.size(); i+=2)
{
   swap(half_edges[i+1], half_edges[i].Pair()); // This is pseudo code, assume this works
}

That's the gist of the idea, merely swap the he right next to you with wherever your pair is.
Now of course, you need to update your pointers, so for both the pair and the HE originally at $i + 1$ you have to go through this tedious process:
    faces[old_pair_face].edge = i + 1;
    verts[old_pair_vert].edge = i + 1;
    edges[old_pair_prev].next = i + 1;
    edges[old_pair_next].prev = i + 1;
    edges[i].pair = i + 1;

    faces[old_other_face].edge = old_pair_index;
    verts[old_other_vert].edge = old_pair_index;
    edges[old_other_prev].next = old_pair_index;
    edges[old_other_next].prev = old_pair_index;
    edges[old_other_pair].pair = old_pair_index;

Essentially just update all the pointers that pointed to the original 2 elements.
BUG
The above ALMOST works. However consider the following case. Assume we are at edge i. The pair of i is j, the next of j is i+1.
So i + 1 and j swap, so the pair of i is now at i + 1. You then want to update next, which points to i + 1, so inadvertently in this specific case, swap will accidentally make next point to yourself. Which is invalid as the Half edge does not allow an edge to point to itself.
This is pretty rare so it took me a while to find, but it's clearly an error in my logic.
The question is, given an already implemented half edge, how can you sort its edges such that the n and n+1 are pairs, while avoiding the pitfalll described above, in O(1) memory?

My solution to this ended up being not sorting at all. Manipualting an already initialized HE is a nightmare, so instead I built the HE with this property from the get go.
The initialization algorithm is pretty much:

Create a new half edge.
Check if the pair of the half edge exists.
If it exists, you are allocated at index of pair + 1
If it does not exist, increment an offset by 2 and store yourself at offset - 2 (i.e. allocate 2 slots in the edge array and store yourself in the even slot).

Doing it this way before setting next and prev and company guarantees that at any one time the HE is properly initialized.


How glVertexAttribPointer works? I know that when you call glVertexAttribPointer, you are specifying how the buffer currently binded should take its own data. But if you can have binded one buffer per type (Let's say GL_ARRAY_BUFFER and GL_ELEMENT_ARRAY_BUFFER) then, Are you specifying two buffers simultaneously? In this tutorial says:

  glVertexAttribPointer always refers to whatever buffer is bound to
  GL_ARRAY_BUFFER at the time that this function is called. Therefore it
  does not take a buffer object handle; it simply uses the handle we
  bound previously.

If glVertexAttribPointer refers to the GL_ARRAY_BUFFER buffer currently binded, How I call glVertexAttribPointer for the GL_ELEMENT_ARRAY_BUFFER buffer?

How I call glVertexAttribPointer for the GL_ELEMENT_ARRAY_BUFFER buffer?

You don't. The element array buffer is used to store vertex indices, not vertex attribute data and therefore it doesn't have attribute pointers. The count, size, and offset for index data is specified in the draw call, e.g. glDrawElements.


What are the current open problems in Computer Graphics? I did a quick investigation about the topic but there doesn't seem a decent resource to find related problems without digging into latest CG papers (unlike CS problems you can easily find a Wikipedia list)

With open problems I mean phenomenas that still do not have a tractable solution/approximation to find its way into CG and better yet real time CG.

There is a vast amount of open problems in real-time graphics: shadows, aliasing, reflections, global illumination, transparencies (blending order and lighting) etc.

SIGGRAPH annually hosts a course called "Open Problems in Real-Time Rendering", which describes current issues in real-time graphics (mostly game dev industry standpoint). You can find last year's materials here: http://openproblems.realtimerendering.com


finding the projection matrix for one point perspective I found the derivation for two point and three point perspective here on this site,  even though it says review on one point perspective,  it  doesn't give a link to previous pages, but I'd like to know how the matrix for one  point perspective was derived. If possible please give me a detailed derivation for this projection matrix.

When reading on transformation matrices over a range of distinct source material, there are some main concepts that you must understand, which are as follows.

Column vs Row Notation

Some authors write vectors as a single row matrix (1x3), while others write them as a single column matrix (3x1). The latter seems to be the preferred representation in most modern books on computer graphics. The main difference between the two notations is that when using the former, matrices will be multiplied on the right side of the vector, and when using the latter they will be multiplied on the left. When dealing with linear transformations, such as rotation matrices, in a particular notation, they are represented as the transposed matrix in the other notation. For instance, the rotation of point $p=(1, 1)$ for an angle $\theta$ in the plane is shown below. As you can easily see, $Row(x)^T = Column(x)$, i.e., one notation is the transpose of the other.

$$
\begin{align}
Row(p) =&
\begin{bmatrix}
1 & 1
\end{bmatrix}
\begin{bmatrix}
\cos(\theta) & sin(\theta)\\
\text{-}sin(\theta) & \cos(\theta)
\end{bmatrix}\\
Column(p) =&
\begin{bmatrix}
\cos(\theta) & \text{-}\sin(\theta)\\
\sin(\theta) & \cos(\theta)
\end{bmatrix}
\begin{bmatrix}
1 \\ 1
\end{bmatrix}
\end{align}
$$

The website you linked uses row notation. This is important because if you read about the perspective matrix in other sources, you may see the result written as the transpose matrix if the author used column notation.

Basis Orientation

In the website, the camera basis vectors $X_1$, $Y_1$, and $Z_1$ form a left-handed coordinate system, because $X_1 \times Y_1 = - Z_1$. This is important as well, as you may notice some sign changes in the perspective matrix if the system was right-handed. You can read more about handedness in Wikipedia. As with row notation, this is another arbitrary decision that the author has taken.

Perspective Projection

Given the appropriate basis, and a projection plane at a distance d from the origin, perspective projection is a simple exercise in similar triangles, as shown below.

The point $\mathbf{P} = (x,y,z)$ when projected onto the plane gives another point $\mathbf{p} = (x_p, y_p, z_p)$ that can be written as (in row notation)
$$\mathbf{p} = \begin{bmatrix}
 d \displaystyle \frac{x}{z} & d \displaystyle \frac{y}{z} & d
\end{bmatrix}.$$

Now we just need to put this formula into matrix form (again, in row notation).

Homogeneous/Projective Coordinates

If we just use the raw 3D coordinates of $\bf P$ there is no way to represent the formula for $\bf p$ as a matrix multiplication. The use of homogeneous coordinates allows us exactly to do that. We start by augmenting $\bf P$ onto the fourth dimension, by defining it as $\mathbf{P} = (x, y, z, 1)$, and we use 4x4 homogeneous matrices instead of 3x3 matrices to represent transformations. I strongly urge you to read on projective spaces, as otherwise the following explanation will look like too much hand-waving as I am not going to dwell on mathematical formalities.

Now, in this new space every point with coordinates $(wx,wy,wz,w)$ can be mapped onto the point $(x,y,z,1)$ by a simple division by w. We can use this property in our advantage to fill-out a matrix in such a way that after the division by w is made, we get exactly the perspective projection formula for our point.

$$
\begin{bmatrix}
x & y & z & \displaystyle\frac{z}{d}
\end{bmatrix}
=
\begin{bmatrix}
x & y & z & 1
\end{bmatrix}
\begin{bmatrix}
1 & 0 & 0 & 0\\
0 & 1 & 0 & 0\\
0 & 0 & 1 & \displaystyle \frac{1}{d}\\
0 & 0 & 0 & 0
\end{bmatrix}
$$

Now if we take the point $(x, y, z, \frac{z}{d})$ and we divide it by its last coordinate, we get $(d\displaystyle\frac{x}{z}, d\displaystyle\frac{y}{z}, d, 1)$, which are exactly the coordinates of our projected point if we just ignore the augmentation onto fourth dimensional space that we did (i.e., discard the last coordinate).

As you can see, the "perspective matrix" does not actually do any perspective transformation, it just "prepares" the transformation by coding a specific value in the last coordinate, such that after all coordinates are divided by it, we get the true projection.


Problem sampling uniformly envmap with GGX I have made a program that samples uniformly an environment map and computes the lighting using the GGX BRDF.
I have made a small example that reproduces the issue I'm having. You can check it out online: shorturl.at/tvzGY
The algorithm works as follows:

We compute V, the unit vector that points from the surface to the camera. FPosition is the position of the fragment in camera space so we can use that to compute V.
We will take a few sample directions L in uniform random directions. These samples are taken inside a hemisphere in the direction of the normal N.
In the real program, we would use L to sample the environment map but, for simplicity, in the provided example, I'm just assuming a complitely white environment (the issue happens anyways).
For each of these samples we compute the rendering equation with the GGX BRDF, and we accumulate that into a variable.
And finally we divide by the number of samples.

I'm using the formulas for GGX I found in this site: http://graphicrants.blogspot.com/2013/08/specular-brdf-reference.html
The problem I'm having is that it doesn't look right. If you open the link you will see this.

That's with roughness = 0.1, metallic = 1.0, and numSamples = 10.
In the top right corner, switch to the sphere. And you will see this.

You see those bright spots, the number of them correlates with the number of samples. That could make sense, maybe. But when I increment the number of samples to 100, we get this.

There are more spots but they are also smaller and blurrier. I think I must be doing something wrong because it doesn't look like a metal. Or is it just because of the uniform sampling? Even with 1000 samples per pixel it doesn't look right to me.

How are you randomizing the samples you take? It looks to me like maybe the random values are the same across all pixels in the image, which would explain why you see 10 coherent bright spots when you have 10 samples.
The samples should be re-randomized for each pixel. This should produce a noisy result, with a single blurred-out bright spot instead of 10 (or however many) coherent bright spots.


Energy Conservation in Lighting Equation I am currently making a Ray-tracer as a personal project. My lighting equation looks like this

Pixel_color[i] = 255 * (Ma*Ig*Ca + Md*I*Cd * diffuse  + Ms*I*Cs * glossy)
// M = material color (ambient, diffuse and specular)
// I = intensity of light (Ig is the global ambient intensity)
// C = color of the light (ambient , diffuse and specular)

Here diffuse and glossy contains all the necessary dot products and the normalization factors ( PI for diffuse and n+8/8*PI for Blinn Phong). Also everything is in range 0-1. I've read from some books and it says to make sure that "Ks + Kd" <= 1 And now I am confused.

1) What exactly is Ks and Kd? Is it just my surface diffuse color and specular color divided by 255 to make it in the range 0-1? Or is it the reflectance ratio that I have to multiply with my color? If it's something else then what should I set them to? For example if I have a color rgb(66, 170, 244) which is somewhat light blue. What should the Ks and Kd be for the above color?

2) These books don't include the ambient component when talking about this conservation so should it be  Ka+Kd+Ks <= 1? Again if that's the case what will Ka be?

3) If I consider Ks = Ms and Kd = Md. It doesn't fit out. for example if I want a surface with the above color rgb(66, 170, 244) to be shiny and the specular highlight should be white. Then my Ms would be (255-66, 255-170, 255-244) which won't be white. Another way I could do is subtract the maximum from every component Ms(255-244, 255-244, 255-244). But this means the specular color would be very low. What if I wanted a very shiny surface?

UPDATE:-

So according to the book Computer Graphics Principles and Practice, Chapter 6 "Fixed Function and Hierarchial Modeling in WPF"

Cd Innate color of “diffuse layer” (Cd,R,Cd,G,Cd,B)
Cs Innate color of “specular layer” (Cs,R,Cs,G,Cs,B)
ka Efficiency of diffuse layer at reflecting ambient light (ka,R, ka,G, ka,B)
kd Efficiency of diffuse layer at reflecting light (kd,R, kd,G, kd,B)
ks Efficiency of specular layer at reflecting light (ks,R, ks,G, ks,B)

After that it writes,

  Note that for solid-color materials, the distinction between the two terms Cd
  and kd is unnecessary, as far as the math is concerned; you can think of them as
  a single term. That is, you can fix kd,R at 1. 0 and use Cd,R to achieve any effect,
  and conversely you could fix Cd,R and specify only kd,R. However, the distinction
  between the two terms is meaningful when the innate color Cd is being provided
  via texture mapping; in that case, there is a need for a kd,R factor affecting the
  reflection of the varying color Cd specified by the texture.

After that in Chapter 14.9 it removes Kd/Ks and Cd/Cs and replaces both with K_L and K_G

  These k parameters are no longer potentially ambiguous RGB triples (each 0 . . . 1), but constants representing the net probability over all directions of
  that term contributing to scattering.

Which led me to think they are just a single constant like "0.5" However in the next example they are telling the same thing to ensure that K_L + K_G <= 1 across all channel which implies that k_L and k_G are infact RGB triples.

Comment: if you make references to book, then it would be good if you could provide references.

To answer your questions:

First, it is quite unusual these days to see people representing colors using a computer byte that is integer values within the range [0:255]. Most people these days use floats, and you will understand why in a moment. When you look at a material in general you can see a shiny part and a diffuse part. Shininess and diffuse appearance are computed using two different types of equation. Let's call the functions that compute the diffuse part diffuse() and the function that compute the specular component specular(). So now you can write:

ObjectColor = Ks * specular() + Kd * diffuse();

Ks and Kd control how shiny or/and diffuse your object is.

About conservation of energy. In short an object can not reflect more light that it receives. So that means that technically if the specular() and diffuse() function do the right thing Ks + Kd ought to be 1. If that wasn't true you would violate the principle of conservation of energy. In practice, it is generally good to follow this rule though most of the basic shading models are based on simple diffuse and specular models (for example the Phong model) and are not themselves energy conserving, so trying to be purely energy conserving at this level, is like trying to say you can paint a miniature with a paint roller. Forget about it, just try to be more or less in the right range.

ks and kd generally stand for the specular and diffuse coefficients.

if you don't do path tracing (aka don't simulate global illumination) the object won't reflect light from other objects in the scene, just light from direct light sources. So in the old time, in order to compensate for that, they would add an ambient term. In these days in age, this is considered as 'heresy'. Though if you account for the ambient term still then Ka + Kd + Ks should be again equal to 1 (same principle).
Not sure to understand what you say here but all these components are additive. You compute the diffuse multiplied by Kd and then you multiply the specular multiplied by Ks and then you add the two resulting numbers together. And that's your object color at the intersection point.

For an introduction to these topics I recommend you read:

Introduction Ray-Tracing

Introduction to Shading

The Phong Model

In order for you to progress, I personally recommend you proceed by step and don't try to understand everything at once. The problem of rendering and shading is very complex. You need to first understand the principle of light-matter interactions, what is diffuse what is specular and how are objects illuminated (direct and indirect lighting). Once you have these notions you can jump onto shading and understand or learn about the concept of shading model, and what the flaws and strengths of existing shading models (many of them exist).


How does UnrealEngine handle depth? Today I found something really interesting (I've never seen something like that). While debugging the buffers used in lightning, I see that the depth information is stored in what looks like cascades of depth.

What's going on there? Is that a technique to improve depth precision?

Thanks!

This is just a visualisation. Type the command vis scenedepthz uv0 to see the actual depth buffer used. UE4 uses a "reversed" depth buffer for the scene.


Rendering fog underneath water I'm aware of glFog, which blends between the fog and surface colour based on distance to the camera, like this:

The equations are even given in the docs.
But the fog is applied immediately in front of the camera. I'm trying to render a water surface and want it to have some depth, for example:

At the moment I'm just rendering the water surface with alpha blending, as below. What I need is to have the fog based on the distance from the water surface to the bottom of the water. How can I get this distance? For the moment I'm assuming the camera will always be above the water.

You can ray trace the distance to the second surface. This may be conceptually easiest, but not necessarily fastest method*.

But there is a neat trick, you can multi pass render a depth map to the water surface from the camera. The distance in water is now pixel depth minus the depth map depth. This does not work in all situations such as when you exit water and re-enter later. But should work in many cases, like the one depicted. You can even use same technique for Boolean operations.

Image 1: Render depth map, consult depth map when shading the bottom of your pond.

* It might be in certain cases.


Why does opengl use 4d matrices for everything? What I managed to figure out is that the first 3*3 is used for rotation and scale and the 1*3 at the end is used for position, but what is the bottom row used for?
is it only for clipping related things?

The bottom row allows you to create perspective foreshortening. That is, it makes lines that are getting further away appear to converge. When arranged this way, we call this a perspective projection matrix.

There are other ways to arrange a projection matrix where that foreshortening doesn't happen. For example in an orthographic projection. This graphic shows some different projections that are possible by  changing the values in the last row of a 4x4 matrix.

Mathematically, the difference between a perspective projection and an orthographic projection is that the last row of a perspective matrix has a division in the z component, whereas the orthographic projection has a 0 in the z component.


How to translate the center of an equirectangular projection? I'm trying to perfectly align two or more equirectangular photos of the same place taken from slightly different positions.
Using an example provided by openMVG I managed to get the relative position between the two shots, but I can't figure out how to translate the center of the projection according to the data obtained.
I tried with Hugin without success, editing the camera position information (it doesn't seem to be able to export the entire equirectangular photo) and I didn't found any useful panotools. I found some literature about the problem but not any real implementation to adapt.

Could you advise me of a possible path to follow?

My requirement is to do it programmatically because I will have to apply it to thousands of photos.

Thanks a lot.

An equirectangular projection treats the x coordinate as the angle theta around the vertical axis going from 0 to 360 degrees. These angles match the longitudinal angles of a globe. The y coordinate is treated as the angle phi that represents the latitudes of a globe. They typically go from +90° at the zenith (or north pole) to -90° at the nadir (or south pole).

So to translate the center of projection of an image by some amount (x0, y0), you  can simply add x0 to the x value of every coordinate (wrapping around to the other side at the edges). The y coordinate is similar - add y0 to every coordinate.  However, instead of wrapping around, you need to reflect the value. So if your new coordinate is less than 0, simply take the absolute value. If it's greater than the height, take the difference between the new value and the height of the image and subtract that from the height of the image. If it was less than 0 or greater than height then you need to also add half the width to the x coordinate.

So putting it in pseudocode:

newY = oldY + translateY;
if (newY < 0)
{
    newY = fabs(newY);
    translateX = fmod((translateX + (width / 2.0)), width);
}
else if (newY >= height)
{
    newY = height - (newY - height);
    translateX = fmod((translateX + (width / 2.0)), width);
}
newX = oldX + translateX;
if (newX < 0)
{
    newX = width + newX;
}
else if (newX >= width)
{
    newX = newX - width;
}


Solid angle and surface area values in (photometric) raytracing So, I started writing my second raytracer - this time focusing on photometric rendering that uses IES lights and standard photometric units.  I've got the basic raytracing up and running (using pure geometry), I've read any article I could found on radio- and photometry (I think I developed an intuition for it) but the issues started when I tried to validate/play with the underlying math.

I use IES lights as a function of spherical coordinates that gives me the intensity of light in that direction in candelas (lumens per unit solid angle). Also in the equation for Luminance/Radiance there are differential solid angle and differential surface area. On the paper everything looks good, however:

My question is: how should I deal with those parameters in my raytracer? I assume (but I'm not sure) that my (differential) solid angle is just a (unit) vector in descrete world and the surface area is simply a point - but what values should I plug in to those equations? I saw that on some websites that people just ignored those terms and replaced the integral with sum - but no explanation was given...

And my second question is related to those IES lights - how can I simply convert intensity to radiance? The intensity already has the solid angle, but for radiance I need the surface area... Or should I somehow integrate the luminious intensity to get luminious flux and then somehow calculate the radiance from that.

EDIT: I decided to try to grab a sheet of paper, try again and share with you my thought process. In order to simplify a lot of things:

each plane has a single scalar value as material (let's name it reflectivity between 0 and 1)

$p1$ has $reflectivity = 0.5$
$p2$ has $reflectivity = 1$

all materials are lambertian (no specular reflections nor emission)

So with those in mind let's imagine that we have following scene:

Step 1 - Illuminance at $p1$

The light luminious intensity at vertical angle $V=0$ is $I = 6619 [cd]$. In order to compute the illuminance at point $p1$ we can create a sphere with radius equal to the distance between the light and intersection point $p1$ which is $r = 2$ meters. That virtual sphere intersects with the first plane creating a small differential area $dA$. Knowing that $d\omega = \frac {dA}{r^2}$ we get $E_L(r) = \frac {I}{r^2} [ \frac {W}{m^2}]$ which is equal to illuminance measured at a plane perpendicular to light. We have to project it to know how many light hits the $p1$, so $E = E_L \times cos \theta$. I assume that $L_i = E$, but that is just a guess. I hope that at this point I got it right...

Step 2 - Outgoing luminance

In order to calculate the outgoing radiance I think that is the place for the rendering equation to kick in:
$$
L_o = L_e + \int_{\Omega} f_r(\omega_i, \omega_o) L_i(p, \omega_i) cos \theta d\omega
$$
since I ignore the emission and my BRDF is $f_r(\omega_i, \omega_o) = \frac {reflectivity}{\pi}$ which is constant across the integral giving me:
$$
L_o = \frac {reflectivity}{\pi} \int_{\Omega} L_i(p, \omega_i) cos \theta d\omega
$$
Now I have to discretize it, which I don't fully understand. Based on my notes that I've made based on many pages I've got:
$L_o = \frac {reflectivity}{\pi} \sum L_i(p, \omega_i) cos \theta \Rightarrow L_o = \frac {0.5}{\pi} L_i cos \theta$, where $L_i$ was calculated in step 1

Step 3 - Luminance at $p2$

Can I assume that $L_{i_{p2}} = L(p_1 \rightarrow p_2) \times cos \theta = L_{o_{p1}} \times cos \theta$ ? I'm not sure if I have to apply the $cos \theta$ twice (for outgoing and incoming $dA$s).

I also would like to apologize for making this question sooo long - but I had never taken any course that would introduce radiometry and I was forced to (try to) learn it by myself.

You have a lot of questions so let me take a shortcut and just try to explain how this is properly done (:

What you are interested in calculating is luminance (cd/m^2) for a pixel. A simple way you can do this is by using brute-force Monte Carlo integration over the hemisphere defined by the normal of a pixel, i.e. cast few thousand randomly distributed rays over the hemisphere. For each of those sample rays you evaluate the BRDF (1/sr) and multiply it with the luminance of the area light source the ray hits, or 0 if it doesn't hit anything. Sum up all these results and multiply by 2*pi/num_samples in the end, i.e. by the solid angle of each sample.

This operation gives you the the luminance for the pixel, which you then multiply by camera exposure factor to map to a proper range which you can tone map, i.e. calculate EV100 from camera aperture, shutter time and ISO-settings as follows:

float ev100=log2((aperture*aperture/shutter_time)*100.0f/iso);

And then calculate exposure factor:

float exposure_factor=1.2f*pow(2.0f, ev100);

For IES light profiles you can normalize the profile and modulate light luminance with the IES factor for the given direction the ray hits the light.


Filling pixels under or above some function Seems like a simple problem, but I just cant wrap my head around it.
I have a config file in which I declare a few functions. It looks like this:
"bandDefinitions" : [
    {
        "0": ["x^2 + 2*x + 5 - y", "ABOVE"]
    },
    {
        "0": ["sin(6*x) - y", "UNDER"]
    },
    {
        "0": ["tan(x) - y", "ABOVE"]
    }
]

These functions should generate 3 images. Every image should be filled depending on solution of equations, and provided position (Under or Above). I need to move the coordinate system to the center of the image, so I'm adding -y into the equation. Part of image which should be filled should be colored white, and the other part should be colored black.
To explain what I mean, I'm providing images for quadratic and sin functions.

What I'm doing is solve the equation for x in [-W/2, W/2] and store the solutions into the array, like this:
#Generates X axis dots and solves an expression which defines a band
#Coordinate system is moved to the center of the image
def __solveKernelDefinition(self, f):
    xAxis = range(-kernelSize, kernelSize)
    dots = []

    for x in xAxis:
        sol = f(x, kernelSize/2)
        dots.append(sol)

    print(dots)
    return dots

I'm testing if some pixel should be colored white like this:
def shouldPixelGetNoise(y, x, i, currentBand):
    shouldGetNoise = True

    for bandKey in currentBand.bandDefinition.keys():
        if shouldGetNoise:
            pixelSol = currentBand.bandDefinition[bandKey][2](x, y)
            renderPos = currentBand.bandDefinition[bandKey][1]
            bandSol = currentBand.bandDefinition[bandKey][0]
            shouldGetNoise = shouldGetNoise and pixelSol <= bandSol[i] if renderPos == Position.UNDER else pixelSol >= bandSol[i]
        else:
            break

    return shouldGetNoise

def kernelNoise(kernelSize, num_octaves, persistence, currentBand, dimensions=2):
    simplex = SimplexNoise(num_octaves, persistence, dimensions)
    data = []

    for i in range(kernelSize):
        data.append([])
        i1 = i - int(kernelSize / 2)

        for j in range(kernelSize):
            j1 = j - int(kernelSize / 2)
            if(shouldPixelGetNoise(i1, j1, i, currentBand)):
                noise = normalize(simplex.fractal(i, j, hgrid=kernelSize))
                data[i].append(noise * 255)
            else:
                data[i].append(0)

I'm only getting good output for convex quadratic functions. If I try to combine them, I get a black image. Sin just doesn't work at all. I see that this bruteforce approach won't lead me anywhere, so I was wondering what algorithm should I use to generate these kinds of images?

Compute the tangent to the line at that point, Use it to make a tangent vector, then use the point on the line, the point being tested and the new point to create two vectors. Take the dot product of the vectors. Positive values are on one side of the line, negative are on the other side. (above and below).


Phong shading in OpenGL: line of light from the center of the world I implemented a Phong shader in GLSL, but there is a bug.

What you are (supposed) to see down below:

A point light source rotating around the center of the world,

with a radiance of (0, 100, 0)

A textured quad with kd of (1, 0, 0).

The problem is:

Why is there a a green line to the center of the world instead of a green spot light where the quad is lit? (where the center of the point light source is)

(Btw in this case there would be only a yellow spot of light, because red + green = yellow)

Vertex Shader:

#version 330

uniform mat4 ModelMatrix, ViewMatrix, ProjectionMatrix;

uniform vec4 wLiPos[5];       // pos of light source
uniform vec3 wEye;         // pos of eye
uniform int lightSize;      //number of lights

layout(location = 0) in vec3 vtxPos;
layout(location = 1) in vec3 vtxNorm;
layout(location = 2) in vec2 vtxUV;

out vec2 texCoord;
out vec3 wNormal;           // normal in world space
out vec3 wView[5];             // view in world space
out vec3 wLight[5];            // light dir in world space

void main() {
    gl_Position = ProjectionMatrix * ViewMatrix * ModelMatrix * vec4(vtxPos, 1);
    texCoord = vtxUV;
    vec4 wPos = ModelMatrix * vec4(vtxPos, 1);
    wNormal = (vec4(vtxNorm, 0) * inverse(ModelMatrix)).xyz;
    for(int i = 0; i < lightSize; i++) {
        wView[i] = wEye * wPos.w - wPos.xyz;
        wLight[i] = wLiPos[i].xyz * wPos.w - wPos.xyz * wLiPos[i].w;
    }
}

Fragment shader:

#version 330

uniform sampler2D samplerUnit;

uniform vec3 kd, ks, ka;   // diffuse, specular, ambient ref
uniform vec3 La, Le[5];    // ambient and point source rad
uniform float shininess;    // shininess for specular ref
uniform int lightSize;

in vec2 texCoord;
in vec3 wNormal;       // interpolated world sp normal
in vec3 wView[5];         // interpolated world sp view
in vec3 wLight[5];        // interpolated world sp illum dir

layout(location = 0) out vec4 fragmentColor;

void main() {
    vec3 color = vec3(0, 0, 0);
    for(int i = 0; i < lightSize; i++) {
        vec3 N = normalize(wNormal);
        vec3 V = normalize(wView[i]);
        vec3 L = normalize(wLight[i]);
        vec3 H = normalize(L + V);
        float cost = max(dot(N,L), 0);
        float cosd = max(dot(N,H), 0);
        color += ka * La + (kd * cost + (ks * pow(cosd, shininess)) * Le[i]) / pow(length(wLight[i]), 2);
    }
    vec3 gamma = vec3(1.0/2.2);
    fragmentColor = vec4(pow(color * vec3(texture2D(samplerUnit, texCoord)), gamma), 1);
}

I forgot to set the wEye uniform, so it was always (0, 0, 0).


RGB nonlinear, what does it mean? What does it mean and why is it not well suited for describing colors by human?

Is CMYK nonlinear also?

In general, non-linear function would mean that an increase in an input does not produce a proportional increase or decrease in output. Mathematically, any function that relates one value to another that doesn't follow a straight line is non-linear. So x raised to the power y is a non-linear function.

This matters for colors because our brains don't perceive light in a linear fashion. If you double the number of photons reaching the retina, the brain doesn't interpret it as twice as bright. Further, we can better distinguish two dark shades that are separated by a certain difference in intensity than two bright shades that are separated by the same fixed intensity difference.

When encoding light values in a computer, we need to choose how precisely to record them without losing too much quality. If we use too many bits, then our hardware becomes more expensive. If we use too few then the image looks bad. Until the last ~10-15 years or so, it was generally considered too expensive to store more than 8 bits per red, green, and blue channel. The problem is that if you encode the light values linearly (where 2x causes your monitor to produce twice as many photons as 1x does), then images stored with only 8 bits per channel will have very obvious banding in bright areas of continuous color.

But remember above that humans can perceive smaller increments in darker shades than lighter? That means that we can use 8 bits per channel, but not use them linearly. If we instead take each 8-bit color channel value and treat it as a value between 0 and 1, if we raise that value to a power then we have smaller steps in the darker areas than in the lighter. This is called gamma-correction or gamma-encoding. We just have to remember to do the reverse (raise the value to the 1 / x power) when we want to output the values on a display.

So if you're dealing with non-linear RGB, it is probably gamma encoded. sRGB is a popular encoding that uses a power of ~2.2. (In truth, it's a little more complicated than just raising it to a power.) So when someone tells you that RGB is non-linear, that's usually what they are referring to. It is entirely possible to store images in linear RGB, so long as you use more than 8 bits per color channel. There are also formats that use a logarithmic encodings, though they are less common outside of film production.

CMYK is more complicated than RGB. Rather than having only a single thing that is outputting the color (the monitor), with CMYK, you have to deal with the medium you're printing on as well as the inks you are printing with. In a typical scenario, you print onto a white surface (such as a piece of paper). Ideally all visible frequencies of light should be reflected off of the paper and into your eyes or camera. When you add ink, the ink absorbs some frequencies of light and reflects others. (And I believe some passes through the ink, hits the paper and is reflected back as well.) The manner in which an ink reflects light is non-linear. Doubling the amount of ink may not cut in half the number of photons reflected back. So CMYK is also non-linear.


Hlsl - registers type Is there any difference between registers types in hlsl?

For example when using register t0 for UAV will it be somehow precache like textures?

According to microsoft's presentation about resource in DX12 and hlsl 5.1 this registers mean nothing, but maybe they only mean that there isn't any real registers like on CPU?

You have to use the appropriate register type for the resource: b registers for constant buffers, t for textures, and u for UAVs. AFAIK, it is not possible to bind a UAV to a texture slot, or otherwise mismatch registers and resources.

However, a resource can have multiple views, so it is possible to have a resource bound as a UAV in one shader and as a regular texture (SRV) in another, which enables all the usual GPU texture sampling features to be used in the latter case.

The details of how the DirectX API concepts of slots, registers, and resources map to the actual hardware are vendor-specific. On one GPU they may literally represent registers of some kind, while on another they may represent a small chunk of metadata in memory somewhere, which describes the resource to the hardware.


Real-time translucency effect While rendering my scene with OpenGL, I sometimes add an overlay which contains information, settings and a few draggable items. Currently, the overlay has a slightly transparent background to make text easily readable while still allowing the scene to shine through and to let the user see the scene behind/through the overlay.

I would like to replace the transparent background with a translucent background, which (as far as I know) requires the clear image of the scene to be heavily blurred*. And especially on mobile devices (e.g. iPad) with high resolutions and limited processing power, lots of texture lookups and real-time rendering don't work well together.

Is there a way to make real-time translucency feasible on mobile devices like the iPad? Or is there a way to avoid the need to heavily blur the scene in every frame?

Edit:

As suggested in the comments, here is an image I just found in the Wikipedia that describes the difference between transparency (right column) and translucency (middle column).

(Image source: Wikipedia)

*I know there is the two-pass blur (first blur in one and then in the other direction) to reduce texture lookups. But for translucency this still requires quite a large number of texture lookups.

I think you may want to take another look at the iOS user interface if you consider real-time blurs to be out of range of mobile hardware:

Blurs are totally in range of mobile hardware. Yes, you need a fairly large number of texture samples for a blur with a large kernel, but the texture samples are also cached very well, and you can use a separable blur, which effectively means that an MxN blur requires M + N samples, not M * N samples. You can also take advantage of hardware bilinear sampling to reduce the number of samples that your blur requires.


Metal API clips all primitives? I am trying to port my simple app from D3D12 to Metal and I am stuck at rendering a simple triangle.
The window does not show any primitives; only the clear color is visible.
Debugging my application, I noticed, that all primitives are clipped (Performance > Primitives > Primitives Clipped: 100%). I fail to understand why this happens, as these are my coordinates:
struct vertex { float x, y, z; };
static const struct vertex vertices[] =
{
    { 0.0f, 0.5f, 0.0f },
    { 0.5f, -0.5f, 0.0f },
    { -0.5f, -0.5f, 0.0f }
};

What I have tried so far:

Explicitly setting the viewport {0, 0, 800, 600, 0, 1}.
Explicitly setting scissors {0, 0, 800, 600}.
Setting the cull mode to MTLCullModeNone.
Setting the fill mode to MTLTriangleFillModeFill.
Changing the winding order.
Altering the vertex input layout; but I am confident, that mine matches the input.

Are there any pitfalls of Metal that I am missing? (i.e. setting the sample mask like in D3D12?)
My pipeline for reference:
    MTLRenderPipelineDescriptor* descriptor = [MTLRenderPipelineDescriptor new];
    descriptor.vertexFunction   = desc->vertex->handle;
    descriptor.fragmentFunction = desc->pixel->handle;
    descriptor.colorAttachments[0].pixelFormat = MTLPixelFormatBGRA8Unorm;

    MTLVertexDescriptor* vs = descriptor.vertexDescriptor;
    vs.attributes[0].format = MTLVertexFormatFloat3;
    vs.attributes[0].offset = 0;
    vs.attributes[0].bufferIndex = 0;

    vs.layouts[0].stride        = sizeof(float) * 3;
    vs.layouts[0].stepFunction  = MTLVertexStepFunctionPerVertex;
    vs.layouts[0].stepRate      = 1;

Note, that I am NOT using MTKView.
The following is my shader used:
struct Out
{
    float4 pos [[position]];
};

vertex Out vertexShader(
    const device float3* vertices [[buffer(0)]],
    unsigned int idx [[vertex_id]]
) {
    Out out;
    out.pos = float4(vertices[idx], 1.0);
    return out;
}

fragment float4 fragmentShader(Out in [[stage_in]])
{
    // TODO: Write fragment shader
    return float4(1, 0, 0, 1);
}

EDIT:
I made following changes and my triangle is finally being rendered:
struct vertex { float x, y, z, w; };
static const struct vertex vertices[] =
{
    { 0.0f, 0.5f, 0.0f, 1.f },
    { 0.5f, -0.5f, 0.0f, 1.f},
    { -0.5f, -0.5f, 0.0f, 1.f }
};

// Vertex Descriptor
    MTLVertexDescriptor* vs = descriptor.vertexDescriptor;
    vs.attributes[0].format = MTLVertexFormatFloat4;
    vs.attributes[0].offset = 0;
    vs.attributes[0].bufferIndex = 0;

    vs.layouts[0].stride        = sizeof(float) * 4;
    vs.layouts[0].stepFunction  = MTLVertexStepFunctionPerVertex;
    vs.layouts[0].stepRate      = 1;

Shader:
struct Out
{
    float4 pos [[position]];
};

vertex Out vertexShader(
    const device float4* vertices [[buffer(0)]],
    unsigned int idx [[vertex_id]]
) {
    Out out;
    out.pos = vertices[idx];
    return out;
}

fragment float4 fragmentShader(Out in [[stage_in]])
{
    // TODO: Write fragment shader
    return float4(1, 0, 0, 1);
}

I don't understand why my application fails, when using 3D vectors instead of 4D ones. Any ideas?

float3 in Metal Shading Language has the same size/alignment as float4, i.e. 16 bytes. I think your original code would have worked using packed_float3 (12 bytes) in the vertex shader.


Rendering part of the scene inside bounding sphere In this video https://youtu.be/0ckE-CZpXAo?t=445, it's suggested to render onto a sphere instead of screen quad to save computation time on lighting calculations. So you create a bounding sphere for each light and render onto it.
But I can't make sense of it. How can we render the scene around the light by rendering it onto a sphere?

How can we render the scene around the light by rendering it onto a sphere?

Well, answer me this: how do you render a scene around a like by rendering a "screen quad"?
We're ultimately talking about deferred rendering, and what is being done is rendering the lighting pass. During the lighting pass, you read data from the gbuffers that tell you that pixel's position, normal, and any other information used by the lighting equation. The texel location you read this data from those gbuffers is just the fragment's location.
The point being made by the video is that the geometry you render to do a lighting pass does not matter. You are only using it to invoke the fragment shader over a region of the image. This is why rendering a full-screen quad works; you're executing the fragment shader over all of the pixels in the screen.
While that works, it does mean you're executing more fragment shader invocations than are strictly necessary for any particular light. Points that correspond to distances outside of the light's useful effect still have to have an FS executed on them.
A sphere is a useful way to pre-cull such fragments. The sphere's radius represents the useful range of effect for a light. Any fragments horizontally or vertically outside of this range don't get produced, since fragments are only produced by rasterizing the sphere. And because the sphere's vertices will be transformed by the same transformation matrix as the scene, and because you will turn on depth tests, you also can cull out cases that are outside of the light's useful range in depth (or at least, if they're closer to the camera than the light's effect).
But there are downsides of this approach. Spheres are complex objects, so there will typically be some overlap in terms of the number of helper fragment shader invocations being executed for a particular pixel. Of greater concern is using relatively high resolution spheres that appear very small on the screen.
Also, you can only execute a single light per FS invocation, instead of being able to bundle lots of lights together. This is especially important for deferred rendering on tile-based renderers.


Is modifying a texture (painting on it) considered a "state change"? The convention in graphics is that performing fewer state changes is better than performing more state changes (switching shaders, binding buffers, binding textures, etc.). For textures, it is faster to render many polygons using a single atlas (for rendering sprites/text) than to individually bind a new texture for each polygon.

Does this hold true if I am continually painting to a texture via glTexSubImage2D? I have a stream of data coming in (over a network) that undergoes processing and is then painted to a texture one row at a time. The data is presented visually in an endless scroll.

Would I be better off painting to one texture rendered on one large rectangle (scrolling the painted data into view)? The idea here is that I would have one (or two) textures bound at any given time while I just continue to paint to it.

Or should I paint lots of little rectangles (only revealing the rectangle when painting is done)? I assume I would bind one texture per rectangle.

Updating an area of memory in the graphics device (a texture, buffer, and the like) is not quite the same as changing a rendering state.

What makes a render state change expensive is the amount of work the driver has to do to validate the new state(s) and reorder the pipeline. This will most likely also incur some synchronization between CPU and graphics device. However, the amount of data transferred between the devices should be small for a state change (probably just a few commands).

For a texture/buffer update on the other hand, the main cost lies in the data transfer itself. In theory, unless you're reading the texture data back to the CPU after the update, there should be no synchronization or pipeline stalls. However, another aspect should be considered: API overhead. Even if the amount of data you're sending to the graphics device is small, if you do that often enough, eventually the cost of communicating with the driver/device will become greater then the cost of the data transfer. That's another reason why batching is so important when optimizing a renderer.

So in your case, the best approach, it seems to me, would be to keep a system-memory copy of the texture that you update whenever new data arrives. Set a dirty flag and consolidate as much updates as possible into one glTexSubImage for the whole texture (or a large sequential portion of it). You can also play with Pixel Buffer Objects and try to do asynchronous data transfer to reduce pipeline stalls as much as possible. If you can implement some kind of double buffering, then you can write to one copy of the texture while the other is being rendered with. This tutorial explores that scenario. That's my intuitive approach, I'd try to reduce the number of API calls and "batch" the texture updates. That being said, this is very speculative, and you'd have to profile and compare that to other approaches, like doing several small updates, to know for sure which is the most performant in your usage case.

As a side note, this presentation by NVidia is also relevant and provides a lot of good insights: Approaching Zero Driver Overhead in OpenGL.


Interpolating vectors on a grid If you have values on a grid and you want to find the value of a point within one of the cells, you can use techniques like bilinear or bicubic interpolation to get the data at that point.

What technique(s) would be used if the data on the grid was a vector (a surface normal specifically)?

I know that quaternions are good for slerping, which I think could be good in this case to do a "bilinear slerp", but quaternions require basis vectors that represent an orientation, not just a single vector.

Also, I'm not sure how you'd use quaternions to apply something like bicubic interpolation.

What methods would be good in this case, to interpolate vectors on a grid?

I did some research and found the answer I was looking for.  The three most common ways to interpolate vectors are:

Slerp - short for "spherical interpolation", this is the most correct way, but is also the costliest.  In practice you likely do not need the precision.
lerp - short for "linear interpolation", you just do a regular linear interpolation between the vectors and use that as a result.
nlerp - short for "normalized linear interpolation" you just normalize the result of a lerp.  Useful if you need your interpolated vector to be a normalized vector.

In practice, lerp/nlerp are pretty good at getting a pretty close interpolated direction so long as the angle they are interpolating between is sufficiently small (say, 90 degrees, or 120 like Simon mentions in his comment), and nlerp is of course good at keeping the right length, if you need a normalized vector.  If you want to preserve the length while interpolating between non normalized vectors, you could always interpolate the length and direction separately.

Here is an example of the three interpolations on a large angle.  Dark grey = start vector, light grey = end vector.  Green = slerp, blue = lerp, orange = nlerp.

Here is an example of a medium sized angle (~90 degrees) interpolating the same time t between the angles:

Lastly, here's a smaller angle (~35 degrees).  You can see that the results of lerp / nlerp are more accurate as the angle between the interpolated vectors gets smaller.

If you do lerp or nlerp, you can definitely do both bilinear as well as bicubic interpolation.

Using slerp, you can do bilinear interpolation, but I'm not sure how bicubic would translate.

I generated these images by taking screenshots from an interactive shadertoy demo I made to demonstrate these differences.  You can see that shadertoy here:
https://www.shadertoy.com/view/4sV3zt


This function syntax is confusing me gladLoadGLLoader((GLADloadproc)glfwGetProcAddress)

I have never seen this syntax before. Presumably glfGetProcAddres is a function, but it has the brackets behind it. Is this only possible in open gl or is my c++ trash.

The signature of glfwGetProcAddress is:

GLFWglproc glfwGetProcAddress(const char *);

It's a function that takes a string and returns a GLFWglproc. which is defined as

typedef void(* GLFWglproc) (void);

Essentially, that means glfwGetProcAddress is a function that takes a string and returns a pointer to function with no parameters and no return value.

GLADloadproc is a type alias defined as follows:

typedef void* (* GLADloadproc)(const char *name);

Essentially, this means that this is the type of a function that takes a string and returns a void pointer (this is a type of pointer that can be safely converted to from every other type of pointer).

in the expression

gladLoadGLLoader((GLADloadproc)glfwGetProcAddress)

Inside, there is the expression

(GLADloadproc)glfwGetProcAddress

Where we have a type in parenthesis followed by an identifier. This is known as a type cast, it means we are taking some value of some type and converting it to some other type.

In this case, we take our GLADloadproc function that takes string and returns pointer to function and we 'disguise' it as a function that takes string and returns void*. This works because we can convert any pointer to void*, even in function pointer return types.

This is all to end up passing it to gladLoadGLLoader, which expects a value of type GLADloadproc.

TL;DR

gladLoadGLLoader takes a GLADloadproc but we want to give it a GLFWglproc (*)(const char *). To be able to do this, we force the type system to take it anyways by using a type cast.


Scale Sampled Depth Value I am currently porting the atmosphere algorithm used by the UE4 engine to my engine.
As the original algorithm from Bruneton is in Kilometers i had to adapt it to my engine which uses meters. Everything is working and its correctly scaled now but i am having some z precision issues on the horizon when drawing the fog. This, i believe, i due to the following math:
float DepthBufferValue = texture(ViewDepthTexture, depthValueUV).r;
vec3 depthClipSpace = vec3(pixPos / vec2(gResolution) * vec2(2.0f) - vec2(1.0f), DepthBufferValue);
vec4 DepthBufferWorldPos = gInvViewProjMat * vec4(depthClipSpace, 1.0);
DepthBufferWorldPos /= DepthBufferWorldPos.w; // perspective division
DepthBufferWorldPos.xyz *= 0.001;

Here, i am basically reading the depth texture, getting the real world position value and scaling it to be in KMs. Due to this scaling (and the inverse multiplication) i am getting some shimmering in the horizon (where the z distance is the greatest)
Since the depth equation is equal to:
 d = a * (1/z) + b

(with a = near, b = far clip and z the distance from camera) is there any mathematical way to scale the DepthBufferValue to a value where the world position would be in kilometers?
In a more practical way, imagining that we have a World Position of (0, 1000, 0) and that gives us a DepthBufferValue of 0.998, can we scale that 0.998 value to the value that we would obtain if we had used the World Position (0, 1, 0) ? (Assuming of course the same near and far clips)
Many thanks!
Edit:
This is the DepthBufferWorldPos outputed to the screen:
https://www.youtube.com/watch?v=kEaU72NWq8I

Algebraically speaking, it's not too hard to modify $d$ in order to scale $z$. If you want to scale $z$ by a factor of $c$, giving $z' = cz$, then
$$
\begin{aligned}
d' &= a \frac{1}{z'} + b \\
&= a \frac{1}{cz} + b \\
(d' - b)c &= a \frac{1}{z} \\
&= d - b \\
d' &= \frac{d - b}{c} + b
\end{aligned}
$$
In short, to scale $z$ by a factor of $c$, you would take the depth value and subtract $b$, scale by $1/c$, then add $b$ back again.
Note that if you use a projection with reversed Z and an infinite far plane, then $b = 0$, which simplifies this to just $d' = d/c$.


Fonts rendered through textures look thinned I'm rendering fonts using Cairo to surfaces, convert those surfaces to RGBA textures and then display them on GL_QUADS using orthotogonal projection and alpha blending. When I use Cairo to render directly to the window (i.e., without hardware acceleration), fonts look perfect. When I show them through OpenGL, they look "thinned". Example: https://imgur.com/a/J7IvG

My texture sizes are powers of two and I do not currently use any scaling (i.e., use the texture coordinates to only show the portion of the texture that actually contains data). I've also tried using the full texture and scaling in the Cairo drawing matrix (i.e., upscaling by Cairo, downloading by OpenGL) which looks even worse.

I believe this has something to do with GL's interpolation, but cannot seem to figure out why it is. When I disable the bilinear GL_LINEAR texture interpolation to GL_NEAREST, they look even worse (and blocky, as if it'd ignore the alpha channel). What can I do to make my font textures look more true to their original?

From looking at your example images this looks like a case of Alpha blending being applied to the RGB channel of the font-atlas texture and then again applied via the A channel when rendering.

The tell-tale sign of this is usually a feint dark halo surrounding your glyphs which is caused by the unwanted mid-greys present in the texture being multiplied by alpha.

For font rendering it is typical to have anti-aliasing in the A channel and have your RGB channels a constant colour.


Compute Shader vs CUDA/OpenCL So I just recently learned about the Compute Shader and it looks from what I have picked up the same idea as parallel programming you would do with CUDA or OpenCL, but in the shader pipeline.

If I want to draw a million cubes in a scene should I be using one method over the other or both. If both how do you split that up so the GPU isn't trying to parallel compute both the shader and another process at the same time

It's not quite correct, today, to think of compute shaders as being "in the shader pipeline" in the same sense that your vertex and fragment shaders are literally hooked up into a pipeline. Compute shaders are not "hooked up" to anything currently, cannot drive rasterization, or directly consume the outputs of rasterization.

What it allows you to do, though, is consume and produce memory resources also used by draw calls in a relatively efficient way. OpenCL makes this rather difficult (see for example https://software.intel.com/en-us/articles/opencl-and-opengl-interoperability-tutorial). This allows you to efficiently integrate compute passes within your renderer, but it is not integrated into the rendering pipeline


Creating a Smooth 3D Mesh from a 2D Outline I am trying to find (or maybe create) an algorithm for creating a smooth, rounded 3D mesh given a 2D outline. I'm trying to recreate a system from a Gamecube game called Amazing Island, and you can see an example of how it works here. A more modern example of what I'm trying to do would be the Soft Edge 3D Doodle tool in Paint 3D.

Basically, you draw a shape with a freeform pen tool, and the shape is "inflated" into a 3D mesh. The result should have smooth, curved, somewhat organic-looking edges, rather than simply extruding the shape into a prism. Each shape can be assumed to be simply connected (meaning shapes with holes through them are not possible), and it can be assumed that the outline won't self-intersect. It would be nice if I could find a way to have non-simply connected shapes, but it's not necessary, and seems like it would be significantly more complicated.

The closest thing that I've been able to find is how to take an outline and triangulate it to make a flat 2D mesh. I'm pretty lost on how I can take that and turn it into what I want though. I can get a list of points representing the vertices of the outline, which seems like it should be the first step.

I'm implementing this in C#, in Unity, but I'm not necessarily looking for code. A description of an algorithm, or even a good idea for how I could approach the problem, would be great. Thank you for your time.

One algorithm that's pretty good for this, but very difficult to implement is to find the Medial Axis of the shape and then have various profiles that are based on the signed distance from the medial axis. You can also use the Straight Skeleton instead of the Medial Axis, but it's also a bit of a pain to calculate.

An easier way that's not quite as accurate, but usually good enough is described here: https://steamcdn-a.akamaihd.net/apps/valve/2007/SIGGRAPH2007_AlphaTestedMagnification.pdf. You can take a bitmap of the shape the user drew, shrink it down to some manageable size, calculate the distance of every pixel from the edges of the shape, then scale it to the size you need (and scale the distances accordingly). Once you have signed distances you can use them with the same 3D profiles mentioned above.

For example, the pillow in the video you showed is basically drawn as a square. Your 3D profile would set the front face height to 0 at the squares edges, and increasing up to some maximum as you moved towards the center, peaking after a short distance and holding up to the center. The back face height would be the same but in the opposite direction.


Sharp Corners with Signed Distance Fields Fonts Signed Distance Fields (SDFs) was presented as a fast solution to achieve resolution independent font rendering by Valve in this paper.

I already have the Valve solution working but I'd like to preserve sharpness around corners. Valve states that their method can achieve sharp corners by using a second texture channels ANDed with the base one, but lacks to explain how this second channel would be generated.

In fact there's a lot of implementation details left out of this paper.

I'd like to know if any of you could point me out a direction to get SDFs font rendering with sharp corners.

EDIT: Please see my other answer with a concrete solution.

I have actually solved this exact problem over a year ago for my master's thesis. In the Valve paper, they show that you can AND two distance fields to achieve this, which works as long as you only have one convex corner. For concave corners, you also need the OR operation. This guy actually developed some obscure system to switch between the two operations using four texture channels.

However, there is a much simpler operation that can facilitate both AND and OR depending on the situation, and this is the principal idea of my thesis: the median of three. So basically, you use exactly three channels (ideal for RGB), which are completely interchangeable, and combine them using the median operation (choose the middle value out of the three).

To accomodate anti-aliasing, we don't work with just booleans, but floating point values, and the AND operation becomes the minimum, and the OR becomes the maximum of two values. The median of three can indeed do both: if a < b, for (a, a, b), the median is the minimum, and for (a, b, b), it is the maximum.

The rendering process is still extremely simple. The entire fragment shader including anti-aliasing can look something like this:

int main() {
    // Bilinear sampling of the distance field
    vec3 s = texture2D(sdf, p).rgb;
    // Acquire the signed distance
    float d = median(s.r, s.g, s.b) - 0.5;
    // Weight between inside and outside (anti-aliasing)
    float w = clamp(d/fwidth(d) + 0.5, 0.0, 1.0);
    // Combining the background and foreground color
    gl_FragColor = mix(outsideColor, insideColor, w);
}

So the only difference from the original method is computing the median right after sampling the texture. You will have to implement the median function though, which can be done with just 4 min/max operations.

Now of course, the question is, how do I build such a three-channel distance field? And this is the tricky part. The most obvious approach that I took in the beginning was to perform a decomposition of the input shape/glyph into three components, and then generate a conventional distance field out of each. The rules for this decomposition aren't that complicated. Firstly , the area with at least 2 out of 3 channels on is the inside. Then, if you imagine this as the RGB color channels, convex corners must be made of a secondary color, and its two primary components continue outward. Concave corners are the inverse: Two secondary colors enclose their common primary color, and the wedge between where both edges continue inward is white. I also found that some padding is necessary where two primary or two secondary colors would otherwise touch to avoid artifacts (for example, in the middle stroke of the "N" in the picture).

The following image is an example decomposition generated by the program from my thesis:

This approach however has some drawbacks. One of them is that the special effects, such as outlines and shadows will no longer work correctly. Fortunatelly, I also came up with a second, much more elegant method, which generates the distance fields directly, and even supports all of the graphical effects. It is also included in my thesis and so is also over a year old. I am not going to give any more details right now, because I am currently writing a paper that describes this second technique in detail, but I will post it here as soon as it's finished.

Anyway, here is an example of the difference in quality. The texture resolution is the same in each image, but the left one uses a regular texture, the middle one uses an ordinary distance field, and the right one uses my three-channel distance field. The performance overhead is only the difference between sampling an RGB texture versus a monochrome one.


Why sending data from gpu to cpu is slower than cpu to gpu? Why sending data from gpu to cpu is slower than cpu to gpu?

I heard the relation is similar to network situation, in detail, the upload is slower than download. However, I could not understand the meaning.

In my opinion,  the sending operation should be same because the two sending buses (cpu to gpu and gpu to cpu) can be made with the same performance (I mean bendwith).

The actual sending of data is the same. The PCIe bus is the same speed in both directions.

However when programming there is a big difference namely that when you send you can immediately start doing something else (including queuing other operations using that new data) while the actual transfer is going on. But when receiving you need to synchronize on the completion of the transfer before you can use the data that was sent.


Phong model: why no multiplication by N dot L in specular term? Classical Phong reflection model, as described on wikipedia, computes diffuse and specular light as follows:
vec3 diffuse = kd * dot(n, l) * i;
vec3 specular = ks * pow(dot(r, v), a) * i;

Where n, l, r, v are the normal, light, reflection and view vectors, kd, ks are surface diffuse and specular color, a is gloss and i is incoming light intensity.
I understand that the diffuse term is such as it is because 1) diffuse reflection is not dependent on viewer direction and 2) surfaces angled away from light recieve less light to scatter, thus the dot(n, l) term.
However, shouldn't the specular term also be multiplied by dot(n, l) for the same reason? Is my assumtion that dot(n, l) computes incoming light wrong?
I know that the Phong model is mostly a hack, but this seems like a simple and performant modification and I haven't found this issue discussed anywhere. I've also looked into the source code of the game Tesseract, which does seem to multiply the specular term by dot(n, l).

The $N \cdot L$ term (I've seen "geometry term" thrown around; not quite accurate, but good enough for here) is part of the rendering equation.  What you're looking at here is an estimate of the inside of the rendering equation—that is, the incoming radiance i times the geometry term dot(n,l) times the BRDF.
The simple answer for why the Phong BRDF doesn't include the geometry term is "Well, it probably should, but the Phong model predates the invention of the rendering equation by more than a decade.".

Phong, B.T.: Illumination for computer generated pictures.  Communications of the ACM 18(6), 311–317 (1975).
Kajiya, J.T.: The rendering equation.  In: Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’86, pp. 143–150.  ACM, New York, NY, USA (1986).

This deficiency was fixed in what is now called the Modified Phong BRDF:

Lewis, R.R.: Making Shaders More Physically Plausible.  Computer Graphics Forum 13(2), 109–120 (1994)

At the end of the day, whether the term is present or not isn't an issue of correctness per-se.  Not having the geometry term when you might expect it just means that the BRDF corresponds to a different surface reflection that doesn't foreshorten much at glancing angles.  Indeed, many other BRDFs have terms that look like reciprocal geometry terms, so that they cancel with the actual geometry term in the rendering equation.
In code, the BRDFs look like:
vec3 f_phong    = rho_s/(TWO_PI/(a+1.0)) * pow(max(0.0,dot(r,v)),a) / dot(n,l);
vec3 f_modphong = rho_s/(TWO_PI/(a+2.0)) * pow(max(0.0,dot(r,v)),a);

And so when multiplied by the light and geometry term in the rendering equation (i.e., i * f * dot(n,l)), you get:
vec3 spec_phong    = i * rho_s/(TWO_PI/(a+1.0)) * pow(max(0.0,dot(r,v)),a);
vec3 spec_modphong = i * rho_s/(TWO_PI/(a+2.0)) * pow(max(0.0,dot(r,v)),a) * dot(n,l);

The obvious should be noted, however: while the Phong BRDF is simple to understand and implement, it is of limited accuracy on real-world materials.  It's not terrible, the way many people think it is, but it does lack important features that can be important on many materials.  It is not the state-of-the-art, and depending on your needs you may like to look elsewhere (GGX microfacet, etc.) for more modern BRDFs.


Do we have OpenGL shaders write out the intermediate *.o-esque file on first compile so that we can skip compiling them in run-time every single time? Shaders get compiled during runtime every single time an app starts. I understand the reasoning for this is that there are so many GPU architectures out there that it does not make sense to ship pre-compiled shaders. However, I wonder if the designers have explored the possibility of storing out the compiled shader during the first run and load the compiled version from the next run. This is akin to having the intermediate "*.o" as in C/C++ compilation and just linking them when we're building the app. If this is possible, is it a bad idea to do this?

OpenGL 4.1's glGetProgramBinary and glProgramBinary exist for this purpose. The first one retrieves a binary that represents the compiled program in its entirety (though none of the state stored within it. It, and its implementation-defined format, can be stored and reloaded via glProgramBinary.

Of course, there are limitations. Implementations are not required to be able to load the format they gave you. That is, a driver change could cause implementations to switch to a different versions, leaving all of your binaries obsolete and un-loadable. So you have to be ready to recompile from the source.

Alternatively, GL 4.6 allows you to employ SPIR-V, an intermediate shader language. The idea here is that you compile your code to SPIR-V, then load that SPIR-V into a shader, where you can move forward from there. There are no version changes that could break your code.

Of course, SPIR-V is an intermediate language, so it still has to be compiled to some degree, along with all of the error checking and the like needed to make the final GPU code. It won't be faster than program binaries, but it's a bit more reliable (for implementations that support it).


How many GPUs can you simultaneously output from using a single PC? I'm aware that it's technically possible to connect a large number of GPUs to a PC for mining purposes https://www.pcgamer.com/asus-has-a-motherboard-that-supports-up-to-19-gpus/, but how many of these can actually be used simultaneously for graphics output?

For example, if I wanted to use these all independently for a CAVE https://en.wikipedia.org/wiki/Cave_automatic_virtual_environment like system, where each GPU will render its own independent view to its own monitor. There should be very little interaction between GPUs. Is there some limitation of the OS, or graphics driver that would prevent me from doing this with more than something like four cards?

To be clear, I'm not interested in anything like SLI here, just using each GPU independently from a single application (think split screen rendering, but each player having their own monitor). I'm currently experimenting with this from Vulkan with two GPUs, but the graphics API isn't important to me if that puts some limit on this. I'm just curious what the upper limit is for this kind of setup.

Here's an older example of the kind of system I have in mind: https://www.jvrb.org/past-issues/9.2012/3511

I don't think there are any specific limits. Each GPU can probably drive three or four monitors, and with multiple GPUs in a system, I don't think there's any barrier to hooking up the maximum number of monitors to each GPU. Here's a video, for instance, detailing a 16-monitor machine with 4 GPUs, on Windows.

Googling around, I did find some people stating that Windows has a 16-monitor or 64-monitor overall limit, but I couldn't verify this from any official documentation. I also saw some claims that Windows has a limit on the overall desktop size that the monitors are virtually placed within, but again could not verify that. In Linux any such limits (if they exist) would naturally be different, and presumably would depend on which window system you're using.

The other potential limiting factor would be CPU performance and PCIe bus bandwidth for keeping all those GPUs fed with data.


Screen Space Ambient Occlusion Issues I'm having problems with Ambient Occlusion. I've tried to follow both John Chapman's improvements over the Crytek AO and LearnOpenGL's tutorial of it when I failed to properly obtain the correct result. http://i.imgur.com/FbNsq9X.png?1

As the camera turns or moves, the screen flashes consistently though it seems arbitrary. Can anyone tell me what I'm doing wrong?

#version 140

in vec2 UV;
in vec4 vRay;
out vec3 SSAOout;

const int kernelSize = 64;

uniform mat4 projection;
uniform sampler2D gbuffer0;
uniform sampler2D gbuffer1;
uniform sampler2D gbuffer2;
uniform sampler2D gbuffer3;
uniform sampler2D noiseTex;
uniform vec3 kernels[kernelSize];

vec4 ViewPosFromDepth(float depth, vec2 TexCoord) {
    float z = depth * 2.0 - 1.0;

    vec4 clipSpacePosition = vec4(TexCoord * 2.0 - 1.0, z, 1.0);
    vec4 viewSpacePosition = invProjMat * clipSpacePosition;
    viewSpacePosition /= viewSpacePosition.w;

    return viewSpacePosition;
}

const vec2 noiseScale = vec2(1366.0/4.0, 768.0/4.0);

const float near = 0.1;
const float far = 100.0;

float LinearizeDepth(float depth)
{
    float z = depth * 2.0 - 1.0; // back to NDC
    return (2.0 * near * far) / (far + near - z * (far - near));
}

void main() {
    vec4 gbuffer0Val = texture(gbuffer0, UV);
    vec4 gbuffer1Val = texture(gbuffer1, UV);
    vec4 gbuffer2Val = texture(gbuffer2, UV);
    vec4 gbuffer3Val = texture(gbuffer3, UV);

    vec3 fragPos    = ViewPosFromDepth(gbuffer0Val.r, UV).rgb;
    float dist      = LinearizeDepth(gbuffer0Val.r);
    vec3 normal     = gbuffer1Val.rgb;
    normal          = decodeNormal(normal);
    vec3 randomVec  = vec3(2 * texture(noiseTex, UV * noiseScale).rg - 1, 0);

    vec3 tangent   = normalize(randomVec - normal * dot(randomVec, normal));
    vec3 bitangent = cross(normal, tangent);
    mat3 TBN       = mat3(tangent, bitangent, normal);

    float bias = 0.025;
    float radius = 3.0;
    float occlusion = 0.0;

    for(int i = 0; i < kernelSize; i++) {
        vec3 sampleKernel = TBN * kernels[i];
        sampleKernel = fragPos + sampleKernel * radius;

        vec4 offset = vec4(sampleKernel, 1.0);
        offset      = projection * offset;    // from view to clip-space
        offset.xyz /= offset.w;               // perspective divide
        offset.xyz  = offset.xyz * 0.5 + 0.5; // transform to range 0.0 - 1.0

        float sampleDepth = texture(gbuffer0, 1-offset.xy).r;
        sampleDepth = LinearizeDepth(sampleDepth);
        float rangeCheck = smoothstep(0.0, 1.0, radius / abs(fragPos.z - sampleDepth));
        occlusion += (sampleDepth >= sampleKernel.z + bias ? 1.0 : 0.0) * rangeCheck;
    }

    occlusion = 1.0 - (occlusion / kernelSize);
    SSAOout = vec3(occlusion);
}

You need to be careful about spaces - position and normal need to be in view space.

To transform depth into view space position I used this method.
So:

vec3 viewPositionFromDepth(vec2 vTexCoord)
{
    float z = texture2D(depthTexture, vTexCoord).r;
    // Get x/w and y/w from the viewport position
    float x = vTexCoord.x * 2.0 - 1.0;
    float y = vTexCoord.y * 2.0 - 1.0;
    vec4 vProjectedPos = vec4(x, y, z, 1.0f);
    // Transform by the inverse projection matrix
    vec4 vPositionVS = invProjection*vProjectedPos;
    // Divide by w to get the view-space position
    return vPositionVS.xyz / vPositionVS.w;
}

To transform normal into view space you need to multiply it by transpose of inverse of view matrix (should be done on CPU):

vec4 n = transpose(inverse(ViewMatrix)) * normalize(texture2D(normalTexture, texCoord)*2.0 - 1.0 );

And in loop while sampling depth with offset to compare with kernel position, this depth should be in same space as position so I did it the same way:

float sampleDepth = viewPositionFromDepth(offset.xy).z;

This is not the most optimal way though, but it worked for me.


Fundamentally, how are 2D bitmaps rendered? Suppose we have a 64-bit word-addressable computer and we want to program it to output a 5x7 character stored as a binary image bitmap (such as the one below) to a memory-mapped display.

Since we have 5 x 7 = 35 pixels per character, we could store a character using 35 bits in a single word. With the least significant bit starting on the left side of the word and with each pixel in the image being represented by the nth bit as shown above, the number "3" above would be stored in memory as: 01110100010000100110000011000101110, followed by 29 unused bits set to 0.

Is this how characters were/are stored in old/modern computers? Or do they use a single byte/word per pixel instead?

If they are stored in this manner, what would the routine in assembly/machine-code (using nothing more than elementary instructions such as bitwise, arithmetic and data transport operations from the computer's Instruction Set Architecture) used to convert this data into an image on the display look like? Would it be something like:

Store the x and y display coordinate for the current pixel to be updated in a certain register.
Store the two chosen RGB values (in this case 0,255,0 for green and 0,0,0 for black) in two other separate registers.
Have two further registers act as counters initialized to 5 and 7 to keep track of the current row and column of the image being rendered.
Test if the column register is not 0. If it isn't, test if the LSB of the bitmap is set to 1, then AND the respective RGB value register with the x and y coordinate register depending on the result, then MOV that result to the display output register.
Decrement the row counter register by 1, test to see if it is 0. If it is, then set it back to 5 and increment the y coordinate by 1 and decrement the column counter by 1.
Shift the register holding the bitmap 1 bit to the left.
JMP to instruction 4.

Is there a simpler or more efficient way to do this? It seems as though even something as simple as rendering a single small text character takes quite a large number of operations and would take around 200 CPU cycles.

Finally, are there any good books or resources on machine-level code for displaying images from scratch, because I haven't been able to find any as they either gloss over this particular subject or the code is written in a high level language or an assembler using macros, all of which are "cheating" and don't explain what is fundamentally going on at the lowest level.

You have to distinguish the text and graphical modes of the graphics board of your machine.

In the old days, mostly the text mode was supported. In this mode the board took care of storing the bitmap definition of the characters and displaying them at the current cursor position. All you had to do was to provide the ASCII code of the character (one byte per character) in a small text buffer.

Nowadays, a high-resolution raster buffer is provided, which is pixel accessible and to which you write color information in some supported format (in the "highest" mode, 3 bytes (RGB) per pixel, for a megapixel or more).

Originally, simple (packed) binary bitmaps of different sizes were used and "blittted" to the raster memory via a request to the device driver, with possible format translation.

Nowadays, characters are mostly defined as vector drawings, which are a resolution-independent description of the outlines, and need to undergo a complex rendering process that includes antialiasing for smooth results.

The rendered output can be cached for fast display, again by blitting.

The overall process is complex, can be hardware accelerated, and is handled by the operating system (GUI management) in a transparent way, together with other graphical primitive drawing operations.


How can I use 3d animation to create a character that uses sign language? I performed a quick search on programs that translate natural language to sign language (through a 3D animation character). Here are examples:

IBM SISI
ProDeaf (which is an android app)

My question is:

how can I make my own character that uses sign language?
Which platforms are preferred for 3D-modeling and animation(e.g. blender, maya, unity, ... etc.)?
Is there specific algorithm(s) to make the character learn the sign language, or it is just a dictionary?

This is quite a broad topic, so I can only give a general overview. If you've never done any interactive animation before, this might be too big a challenge to start with, and maybe you should think of something smaller to do first.

You first need to model and rig your character, or try to find an existing model online that's rigged with enough control over the fingers. You then need to define a "dictionary" of animations or poses for each item of vocabulary in the language. To create the poses, it is probably easiest to use motion-capture technology to record a proficient sign-language speaker.

All of the above needs to be done in a 3D modelling and animation tool. Maya is very popular for character animation.

To get natural sign-language, you'll also need to write an algorithm to understand the way places in space are used as pronouns, and a fall-back that spells things out using letters when you don't have a vocab item for them. Based on next to no knowledge of sign language, I'd guess the algorithm will end up a little more complicated than a normal text-to-speech algorithm.

You've already mentioned Unity 3D in your question, and that might be a good choice for the real-time part of the system (assuming you want to render the character in real-time). If you use that, you can write the algorithm in C#. You would import the animation/pose dictionary into your system in Unity, and have your algorithm compute the transitions from one pose or animation to the next. In Unity itself, you can define the camera and lighting layout for your model and the backdrop to draw it in front of.


How to make a translation relative to the line ax + by + c = 0, affine transformations (sorry for my english)
The task:
A triangle is specified. To realize its motion on the basis of mirror reflection with respect to an arbitrary line ax + by + c = 0, the coefficients of which are entered by the user.

Already done:
I read from the fields on the form the coordinates of the three points of the triangle.
Putting them into objects of class Point (left (x, y), top (x, y), right (x, y)).
Make an array of points from the points of a triangle:

Point[] points = new Point[3] { left, top, right };

Draw a coordinate system:

DrawField();
DrawCoordinateSystem();

Draw a triangle:

graph.FillPolygon(Brushes.ForestGreen, points);

Make a mirror reflection of the triangle:

Point[] pointsCopy = (Point[]) points.Clone();
for (int i = 0; i  pointsCopy.Length; i++)
    pointsCopy[i].Y = -pointsCopy[i].Y;
graph.FillPolygon(Brushes.RoyalBlue, pointsCopy);

Read the coefficients (a, b, c) from the fields on the form for the equation ax + by + c = 0.

Question:
Tell me how to move it further relative to the line ax + by + c = 0 ?

To mirror an object in relation to an arbitrary line, you first have to find the coordinates of that object in the frame of reference of that line. For convenience, we will define a frame of reference where the $X$ axis is colinear with the line itself and the $Y$ axis will be perpendicular to the line. Then, we can flip the Y coordinate in that frame of reference to obtain the mirrored point and finally we will reverse the transform from the line coordinate system and back to the global coordinate system.

To define a coordinate system, we will need to use one point the line passes through as the origin $O$ of that coordinate system. This can be obtained easily by setting $x = 0$ or $y = 0$ in the line equation and we end up with the point $\{-c/a, 0\}$ or $\{0, -c/b\}$. Getting the $X$ and $Y$ axis of the new coordinate system is easy. The $X$ axis vector $\{X_x, X_y\}$ will be computed by normalizing $\{a, b\}$ from the line equation. Then we need a perpendicular vector for the $Y$ axis direction. This is easily obtained by using either $\{-X_y, X_x\}$ or $\{X_y, -X_x\}$ (you can verify that by taking the dot product between $X$ and $Y$ which will be zero).

Now that you have an origin and a coordinate system, let's find a mirrored point of vector P on that system. First we need to express it in the new coordinate system. This can be done by projecting $P-O$ to the basis vectors $X$ and $Y$, so the coordinates in the new system will be $\{\langle(P - O), X\rangle, \langle(P - O), Y\rangle\}$.
To obtain the mirrored point, we simply flip the Y coordinate. To obtain the final vector in the original space, we simply invert the first transform. The final point should be $O + X\cdot\langle(P-O), X\rangle - Y\cdot\langle(P-O), Y\rangle$

A note on notation: $\langle u,v\rangle$ was used for dot product between vectors. $u\cdot k$ was used for product between vector and scalar