How to unproject cursor with orthographic projection With perspective projection we can unproject screen space coordinates of the cursor to the near and far plane of the frustum and calculate the direction of the ray through the cursor.
        Vector4 cScreen0 = Vector4(cursorNormX, cursorNormY, -1, 1);
        Vector4 cView0 = Inverse(projection)*cScreen0;
        cView0 = cView0*(1/cView0.w);
        Vector4 cWorld0 = Inverse(view) * cView0;

        Vector4 cScreen1 = Vector4(cursorNormX, cursorNormY, 1, 1);
        Vector4 cView1 = Inverse(projection)*cScreen1;
        cView1 = cView1*(1/cView1.w);
        Vector4 cWorld1 = Inverse(view) * cView1;

        Vector3 cursorDir = normalize(cWorld1.xyz()-cWorld0.xyz());

Now however with orthographic projection the far and near plane are of the same size, so we can't calculate the direction of the cursor this way. The direction is going to be equal the world's z axis. (I haven't had much sleep so I hope this makes sense).
So instead I calculated the cursor position by unprojecting the cursor with zeroed out z value. We can calculate the x and y coordinate and set the z coordinate later as we like.
        Vector4 cScreen = Vector4(cursorNormX, cursorNormY, 0, 0);
        Vector4 cView = Inverse(projection)*cScreen;
        cView = Vector4(cView.x, cView.y, 0, 0);
        Vector4 cWorld = Inverse(View) * cView;
        cWorld = Vector4(cWorld.x, cWorld.y, 0, 0);

        Vector3 cursorPos = cWorld.xyz();

However I'm not getting the correct results from the projection. What am I missing?
The purpose of this is to be able to cast rays in the direction of the cursor.

I am still not 100% sure, if I understood your question, because of this sentence:

Now however with orthographic projection the far and near plane are of the same size, so we can't calculate the direction of the cursor this way. The direction is going to be equal the world's z axis. (I haven't had much sleep so I hope this makes sense).

If I misunderstood you, let me know in the comments and I will adjust or remove my answer.
However, if I understood your intention correctly and you want to cast a ray through your frustum (for example to pick objects), then your statement is wrong. The direction will be equal to the view spaces' negative z-direction, not the world spaces'. So all you need to do is to transform your direction vector or the near and far plane points to world space. To proof, that this works, I have implemented everything in a Python script, that you will find at the end of this answer. If you have a Python interpreter with MatPlotLib and NumPy installed, you can modify the setup parameters and experiment a little bit yourself.
So let's have a look at the relevant implementation. First, we calculate the mouse position in clip space and the corresponding 2 points on the near and far plane.
mouse_pos_x_clip = mouse_pos_x_screen / screen_width * 2 - 1
mouse_pos_y_clip = mouse_pos_y_screen / screen_height * 2 - 1

mouse_pos_near_clip = np.array([mouse_pos_x_clip, mouse_pos_y_clip, -1, 1], dtype=float)
mouse_pos_far_clip = np.array([mouse_pos_x_clip, mouse_pos_y_clip, 1, 1], dtype=float)

Now we get the involved matrices. My notation here is as follows: I use two characters after M_ that are abbreviations of the involved spaces. The first character is the source and the second the target space. The characters are c for clip space, v for view space, and w for world space. So M_vc is the view space to clip space transformation a.k.a the projection matrix.
M_wv = get_world_to_view_matrix(camera_pitch, camera_yaw, camera_position)
if perspective:
    M_vc = get_perspective_mat(field_of_view, z_near_plane, z_far_plane, aspect_ratio)
else:
    M_vc = get_orthogonal_mat(frustum_width, frustum_height, z_near_plane, z_far_plane)

M_vw = np.linalg.inv(M_wv)
M_cv = np.linalg.inv(M_vc)

Now I simply use the correct transformation matrices to transform from clip to world space. Note that the perspective projection needs a division by w after the transformation to view space. This is not necessary for the orthographic projection, but performing it does not affect the result.
mouse_pos_near_view = np.matmul(M_cv, mouse_pos_near_clip)
mouse_pos_far_view = np.matmul(M_cv, mouse_pos_far_clip)

if perspective:
    mouse_pos_near_view= mouse_pos_near_view / mouse_pos_near_view[3]
    mouse_pos_far_view= mouse_pos_far_view / mouse_pos_far_view[3]

mouse_pos_near_world = np.matmul(M_vw, mouse_pos_near_view)
mouse_pos_far_world = np.matmul(M_vw, mouse_pos_far_view)

This is, as far as I can see, identical to your first code section. Now let's have a look a the result for perspective and orthographic projection with the following setup parameters:
screen_height = 1080
screen_width = 1980

mouse_pos_x_screen = 500
mouse_pos_y_screen = 300

camera_position = [3, 0, 1]
camera_yaw = 20
camera_pitch = 30

z_near_plane = 0.5
z_far_plane = 3

# only orthogonal
frustum_width = 3
frustum_height = 2

# only perspective
field_of_view = 70
aspect_ratio = screen_width / screen_height

The screen space and clip space values are identical for both projections:

The red line connects the two points on the near and far plane. The red dot is the point on the near plane, which is your "screen" in a 3d space. The green lines mark the borders of the frustum. In clip space, it is obviously just a cube. An important thing to realize is, that clip space is defined in a left-handed coordinate system while the other coordinate systems are usually right-handed (have a look at the images in this link). I mention it since I had some problems with the plots until I realized that.
Now for perspective projection, I get the following plots:

The blue dot is the camera position.
If I just exchange the perspective matrix with an orthographic projection matrix, the results look like this:

As you can see, the approach you used in your first code section works independently of the chosen projection. I don't know why you thought it wouldn't. An assumption of mine is, that you might have made a small mistake during the implementation of the orthographic projection matrix. For example, if you accidentally flipped rows and columns (transposed) of the orthographic projection matrix, you get total crap like this:

I know this looks like a wrong implementation of the perspective projection, but this is what I get when I transpose the orthographic projection matrix before the multiplication.
So make sure you use the correct orthographic projection matrix (source):
$$
\begin{bmatrix}
\frac{2}{w}&0&0&0\\
0&\frac{2}{h}&0&0\\
0&0&\frac{-2}{f-n}&-\frac{f+n}{f-n}\\
0&0&0&1
\end{bmatrix}
$$
Here $w$ is the frustum width, $h$ the frustum height, $f$ the far plane z-value and $n$ the near plane z-value. This is the representation if you use column vectors and left multiplied matrices. For row vectors and right multiplied matrices, you need to transpose it.
Your second approach:
Vector4 cScreen = Vector4(cursorNormX, cursorNormY, 0, 0);
Vector4 cView = Inverse(projection)*cScreen;
cView = Vector4(cView.x, cView.y, 0, 0);
Vector4 cWorld = Inverse(View) * cView;
cWorld = Vector4(cWorld.x, cWorld.y, 0, 0);

Vector3 cursorPos = cWorld.xyz();

has multiple issues and all are related to the z- and w-components of your vectors. Basically, you need to do the same transformations as in your first approach. So use Vector4 cScreen = Vector4(cursorNormX, cursorNormY, -1, 1); as initial vector.
One problem of the line cView = Vector4(cView.x, cView.y, 0, 0); is, that your z-component should be identical to your near plane value and not zero. You might get away with that since it would just shift your point a little bit in the camera viewing direction in world space, but more problematic is that you set w to 0. This makes it impossible to apply any translation to the vector by $4 \times 4$ matrix multiplication. So when you transform to world space, you will always end up with a point that treats the camera to be located at the coordinate system origin, regardless of its true position. So you need to set the w-component to 1. However, if the previous lines are correct, you should automatically get the correct z- and w-values which makes this line obsolete.
Lastly, the line cWorld = Vector4(cWorld.x, cWorld.y, 0, 0); doesn't make much sense either to me. Your camera is somewhere in 3d world space. Why do you remove the z-component you previously calculated? With this, you move the point into the XY-plane for no reason. Just remove this line.
To get the cameras viewing direction without using the far plane point, just multiply the vector [0, 0, -1, 0] with the view-to-world matrix (M_vw). In this case, the w-component really has to be 0, since you do not want to apply translations to it because it is a direction vector and no point. The z-component needs to be -1 because the camera looks into the negative direction by definition. Notice, that the transformed vector is usually not of unit length anymore. So you might want to normalize it.
Additional Note
For an orthographic projection, there is no need to calculate the inverse projection matrix. You can simply calculate the x and y values directly by something like this (untested pseudo-code):
x_view = (x_screen / screen_width - 0.5) * frustum_width
y_view = (y_screen / screen_height - 0.5) * frustum_height

Then you get the screen space point in view space by setting (untested pseudo-code):
point_view_near = [x_view, y_view, -z_near, 1]

Watch out to use the negative near plane z-value! For the far plane, you can do the same.
Full Python script
import numpy as np
from mpl_toolkits.mplot3d import Axes3D  # noqa: F401 unused import
import matplotlib.pyplot as plt

# setup --------------------------------------------------------------------------------

screen_height = 1080
screen_width = 1980

mouse_pos_x_screen = 500
mouse_pos_y_screen = 300

camera_position = [3, 0, 1]
camera_yaw = 20
camera_pitch = 30

# ----------------
# projection setup
# ----------------
perspective = False # set 'False' for orthogonal and 'True' for perspective projection

z_near_plane = 0.5
z_far_plane = 3

# only orthogonal
frustum_width = 3
frustum_height = 2

# only perspective
field_of_view = 70
aspect_ratio = screen_width / screen_height

# functions ----------------------------------------------------------------------------

def render_frustum(points, camera_pos, ax, right_handed=True):
    line_indices = [
        [0, 1],
        [0, 2],
        [0, 4],
        [1, 3],
        [1, 5],
        [2, 3],
        [2, 6],
        [3, 7],
        [4, 5],
        [4, 6],
        [5, 7],
        [6, 7],
    ]
    for idx_pair in line_indices:
        line = np.transpose([points[idx_pair[0]], points[idx_pair[1]]])
        ax.plot(line[2], line[0], line[1], "g")
    if right_handed:
        ax.set_xlim([-5, 5])
    else:
        ax.set_xlim([5, -5])
    ax.set_ylim([-5, 5])
    ax.set_zlim([-5, 5])
    ax.set_xlabel("z")
    ax.set_ylabel("x")
    ax.set_zlabel("y")
    ax.plot([-5, 5], [0, 0], [0, 0], "k")
    ax.plot([0, 0], [-5, 5], [0, 0], "k")
    ax.plot([0, 0], [0, 0], [-5, 5], "k")
    if camera_pos is not None:
        ax.scatter(
            camera_pos[2], camera_pos[0], camera_pos[1], marker="o", color="b", s=30
        )

def render_ray(p0,p1,ax):
    ax.plot([p0[2], p1[2]], [p0[0], p1[0]], [p0[1], p1[1]], color="r")
    ax.scatter(p0[2], p0[0], p0[1], marker="o", color="r")

def get_perspective_mat(fov_deg, z_near, z_far, aspect_ratio):
    fov_rad = fov_deg * np.pi / 180
    f = 1 / np.tan(fov_rad / 2)

    return np.array(
        [
            [f / aspect_ratio, 0, 0, 0],
            [0, f, 0, 0],
            [
                0,
                0,
                (z_far + z_near) / (z_near - z_far),
                2 * z_far * z_near / (z_near - z_far),
            ],
            [0, 0, -1, 0],
        ]
    )

def get_orthogonal_mat(width, height, z_near, z_far):
    r = width / 2
    t = height / 2

    return np.array(
        [
            [1 / r, 0, 0, 0],
            [0, 1 / t, 0, 0],
            [
                0,
                0,
                -2 / (z_far - z_near),
                -(z_far + z_near) / (z_far - z_near),
            ],
            [0, 0, 0, 1],
        ]
    )

def get_rotation_mat_x(angle_rad):
    s = np.sin(angle_rad)
    c = np.cos(angle_rad)
    return np.array(
        [[1, 0, 0, 0], [0, c, -s, 0], [0, s, c, 0], [0, 0, 0, 1]], dtype=float
    )

def get_rotation_mat_y(angle_rad):
    s = np.sin(angle_rad)
    c = np.cos(angle_rad)
    return np.array(
        [[c, 0, s, 0], [0, 1, 0, 0], [-s, 0, c, 0], [0, 0, 0, 1]], dtype=float
    )

def get_translation_mat(position):
    return np.array(
        [
            [1, 0, 0, position[0]],
            [0, 1, 0, position[1]],
            [0, 0, 1, position[2]],
            [0, 0, 0, 1],
        ],
        dtype=float,
    )

def get_world_to_view_matrix(pitch_deg, yaw_deg, position):
    pitch_rad = np.pi / 180 * pitch_deg
    yaw_rad = np.pi / 180 * yaw_deg

    orientation_mat = np.matmul(
        get_rotation_mat_x(-pitch_rad), get_rotation_mat_y(-yaw_rad)
    )
    translation_mat = get_translation_mat(-1 * np.array(position, dtype=float))
    return np.matmul(orientation_mat, translation_mat)

# script -------------------------------------------------------------------------------

mouse_pos_x_clip = mouse_pos_x_screen / screen_width * 2 - 1
mouse_pos_y_clip = mouse_pos_y_screen / screen_height * 2 - 1

mouse_pos_near_clip = np.array([mouse_pos_x_clip, mouse_pos_y_clip, -1, 1], dtype=float)
mouse_pos_far_clip = np.array([mouse_pos_x_clip, mouse_pos_y_clip, 1, 1], dtype=float)

M_wv = get_world_to_view_matrix(camera_pitch, camera_yaw, camera_position)
if perspective:
    M_vc = get_perspective_mat(field_of_view, z_near_plane, z_far_plane, aspect_ratio)
else:
    M_vc = get_orthogonal_mat(frustum_width, frustum_height, z_near_plane, z_far_plane)

M_vw = np.linalg.inv(M_wv)
M_cv = np.linalg.inv(M_vc)

mouse_pos_near_view = np.matmul(M_cv,mouse_pos_near_clip)
mouse_pos_far_view = np.matmul(M_cv,mouse_pos_far_clip)

if perspective:
    mouse_pos_near_view= mouse_pos_near_view / mouse_pos_near_view[3]
    mouse_pos_far_view= mouse_pos_far_view / mouse_pos_far_view[3]

mouse_pos_near_world = np.matmul(M_vw, mouse_pos_near_view)
mouse_pos_far_world = np.matmul(M_vw, mouse_pos_far_view)

# calculate view frustum ---------------------------------------------------------------

points_clip = np.array(
    [
        [-1, -1, -1, 1],
        [ 1, -1, -1, 1],
        [-1,  1, -1, 1],
        [ 1,  1, -1, 1],
        [-1, -1,  1, 1],
        [ 1, -1,  1, 1],
        [-1,  1,  1, 1],
        [ 1,  1,  1, 1],
    ],
    dtype=float,
)

points_view = []
points_world = []
for i in range(8):
    points_view.append(np.matmul(M_cv, points_clip[i]))
    points_view[i] = points_view[i] / points_view[i][3]
    points_world.append(np.matmul(M_vw, points_view[i]))

# plot everything ----------------------------------------------------------------------

plt.figure()
plt.plot(mouse_pos_x_screen,mouse_pos_y_screen, marker="o", color="r")
plt.xlim([0, screen_width])
plt.ylim([0, screen_height])
plt.xlabel("x")
plt.ylabel("y")
plt.title("screen space")

plt.figure()
ax_clip_space = plt.gca(projection="3d")
render_ray(mouse_pos_near_clip, mouse_pos_far_clip, ax_clip_space)
render_frustum(points=points_clip, camera_pos=None, ax=ax_clip_space, right_handed=False)
ax_clip_space.set_title("clip space")

plt.figure()
ax_view = plt.gca(projection="3d")
render_ray(mouse_pos_near_view, mouse_pos_far_view, ax_view)
render_frustum(points=points_view, camera_pos=[0, 0, 0], ax=ax_view)
ax_view.set_title("view space")

plt.figure()
ax_world = plt.gca(projection="3d")
render_ray(mouse_pos_near_world, mouse_pos_far_world, ax_world)
render_frustum(points=points_world, camera_pos=camera_position, ax=ax_world)
ax_world.set_title("world space")

plt.show()


What happens with the framebuffer after the fragment shader is done? I'm wondering what happens with the framebuffer between the time that the fragment shader is done, and the time when it appears on my screen.

Is my understanding correct if I assume that the framebuffer is in memory on the GPU, which is then copied back to the CPU side after glDrawX is completed, and then fit to a window somewhere on the screen, and then sent through my HDMI cable (via the GPU again!) to my monitor?

It entirely depends what hardware and OS you're on. It's up to the integration between the windowing system and GL. In the simple case where one application is rendering full-screen directly, the framebuffer is in memory that's accessible by both the GPU and the display controller. After the GPU is finished, and when the display controller gets to a vsync (the time to start drawing a new frame to the screen), the display controller just reads out the framebuffer.

In a compositing system like modern MS Windows or Android, your app's framebuffer is not used by the display controller directly. After you finish drawing your frame, the compositor runs. On the GPU, it reads in your framebuffer, puts it in the right place on the desktop, and writes the composited desktop into the real framebuffer. The display controller then reads out the desktop framebuffer. There's also an optimization possible using overlays, which we have another question about.

Copying completed frames into CPU memory is to be avoided. If you had to do this every frame, your application probably couldn't run at a reasonable frame-rate. Even reading back a single pixel (with glReadPixels) is a big slow-down, because the CPU has to wait for the GPU to finish, and they have to invalidate the cache of any shared memory. Reading back a million pixels is extra slow on desktop because the CPU and GPU have separate memory and the path between them is relatively slow. It's very rare that it should be necessary: mainly it's needed for screenshots, or for debugging.


Volumetric integration in Frostbite In their presentation at SIGGRAPH 2015 Electronic Arts presented method of volumetric rendering using 3D texture aligned with view frustum.

This technique needs two steps:

Injection of color and density into 3D texture,
Integration along Z axis of texture.

Integration code example is on slide 27. However, it is stated as being wrong. Do anyone can explain what is wrong with that (pretty standard raymarching integration) code?

What is wrong with it seems to be explained on the next two slides: it leads to non-energy-conserving results, where the apparent brightness of the volume changes depending on the scattering coefficient.

My read of it is that the "wrong" integration code implicitly assumes a constant transmittance over the extent of each voxel. But the trouble is, when the scattering coefficient is high, the transmittance will fall off significantly within a single voxel. For instance, if you have voxels of size 10 cm, but the scattering coefficient is 1 / (5 cm), then the transmittance falls off by exp(10 cm / 5 cm), or about a factor of 7.4 over the length of the voxel!

Therefore, according to slide 28, a better approach is to explicitly account for the transmittance falloff over the voxel length, while still assuming constant inscattered light. Fortunately the integral has a simple closed-form solution. So, they're saying to evaluate that formula per voxel in the integration loop, in place of the former "wrong" one.

For completeness: in the code from slide 27, replace this line:

accumScatteringTransmittance.rgb += scatteringExtinction.rgb *
    accumScatteringTransmittance.a;

with this:

accumScatteringTransmittance.rgb += scatteringExtinction.rgb *
    accumScatteringTransmittance.a *
    ((1.0 - transmittance) / scatteringExtinction.a);

Here, accumScatteringTransmittance.a is the transmittance at the point the ray enters the voxel (i.e. the resultant transmittance of all the voxels in front of it), and then the ((1.0 - transmittance) / scatteringExtinction.a) factor accounts for the varying transmittance within the voxel.


How to get polygon coordinates in screen space I need to be able to click on this polygon in 2d. This polygons set of vertices aren't in the same space as the screen. (Ie: the edge of the screen may be a value of 20 in one space but 1920 in the other) the scale is different. How can I convert the vertices of the polygon to the screen space so that I can use the mouse coordinates to click on it. (This is only 2d)

Edit: This answer is only helpful in 3D

If you want to do it geometrically...

The inverse of the view-projection matrix, $K^{-1}$ is the matrix you want. Where $K = View * Projection$

If $\vec{v}$ is a point in homogeneous screen coordinates, $[x, y, 1]$ where $x$ and $y$ are whatever coordinates your $K$ matrix projects onto.

$$\vec{d} = K^{-1} * \vec{v}$$

Where $\vec{d}$ now represents a line in homogeneous coordinates, coming out of your camera origin. Now you'd iterate through the objects in view, and choose the closest one in front of the camera that this line intersects with.

But there are other ways!

Color Picking

Here, we render (into a buffer) each object in view with a slightly different color, which corresponds to the object ID. We then read it using glReadPixels(mouse_coordinates...) and decide which object we clicked. This is pretty easy to do in the new pipeline - you just write a color passthrough shader. Would not recommend pre OpenGL 3.0 (You'll have to do shenanigans to render the scene without interference)

Selection Buffer (Old pipeline)

If you're on the fixed pipeline, you get a nice little toy called the selection buffer to do the work for you.


Are the terms screen and the monitor in computer graphics used interchangeably? I was looking at this Wikipedia article about the image plane or the screen space

  https://en.wikipedia.org/wiki/Image_plane

and at a certain point there's

  A rectangular region of this plane, called the viewing window or viewport, maps to the monitor.

Here, by "plane" they are (I think) referring to the image plane (or screen space); but I'm wondering now what do they mean here by monitor.

Is the monitor the same thing as the screen? Is the screen the plane in the monitor, but people in computer graphics usually say only monitor to refer to the screen?

Or, maybe first of all, what is intended by screen and monitor in computer graphics?

The terms screen, monitor, and display are used more or less interchangeably to refer to the real-world physical device that shows the image you're looking at. (They're not 100% interchangeable: monitor means a discrete display device such as you'd use with a desktop PC. However, a built-in display such as the one in a laptop, tablet, or phone is never called a monitor.)

The image plane means the abstract geometric plane that identifies the surface of the display, but extends infinitely in all directions, i.e. isn't bounded by the edges of the display. The viewport or viewing window is the rectangle within this plane that identifies the edges of the display (or more generally, the edges of the image you're rendering, in case your app is running in a window and not fullscreen).

The term screen might also be used more colloquially to mean various other things, such as: various coordinate systems on the image plane ("screen space"), the viewport or view frustum ("the object is off-screen"), the back buffer ("render it to the screen"), and probably more.


Special Texture Stretching without deformation I am trying to achieve a special texture stretching effect in my shader.
Here is how I define my uv before sampling my texture, nothing really special.

// uv_ST contains the tiling and offset values
uv.xy = uv.xy * uv_ST.xy + uv_ST.zw

This gives the standard tiling/stretching and offset behaviour when you tile/stretche a clamped texture as you can see in the image below. First is normal, second is offset and last is stretching.

But I want to avoid the deformation behaviour when stretching, I want to keep margins when stretching my texture or simply cut it in the middle and stretching it. Here is an illustration below.

How could I do that inside my shader when defining my uvs before sampling the texture ?

The Nine Patches algorithm is a really nice and powerful solution if you are using textures but I ended up by using a fully procedural solution.

I am just drawing a round box like this.

float udRoundBox( vec3 p, vec3 b, float r )
{
  return length(max(abs(p)-b,0.0))-r;
}

You can simply avoid the 3rd dimension if you don't need it.
This method comes from the Distance function article from Inigo Quilez.


Do animation studios like Pixar, Dreamworks, and Disney use super-sampling? Does any of their rendering techniques involve rendering an area at a higher resolution just to scale it down in order to mitigate artifacts? I know they use ray methods rather than rasterization however ray methods still have artifacts even when done with a ton of rays.. right?

I noticed a few moments in the film Moana that seemed to be poorly rendered because small details seemed fuzzy and sorta pixelated. Especially around the edges of a figure that was in focus with a slightly blurred background. Could they have used supersampling on those scenes to decrease the issue?

Super-sampling for ray-tracing doesn't just mean rendering an area at higher resolution and then downscaling the finished image: in fact, this simplest super-sampling technique is really not effective enough for high-quality production rendering.

In a production ray-tracer, super-sampling means sending more than one primary ray per pixel (16 primary rays per pixel is a good average) and then using signal reconstruction techniques (i.e. not just averaging) to compute a colour for each pixel of the image. Using lots of primary rays doesn't just help to avoid aliasing (jaggies) from when polygons partially cover pixels; it also helps to smooth over noise added in shading, such as when sampling global illumination or fuzzy reflections.

In addition, super-sampling by sending more rays lets you jitter the sample positions to avoid the artefacts you get from regular sampling. It also lets you use the results of previously traced primary rays to decide where to put more samples.

For example, the simplest adaptive sampling algorithm is contrast based: within each pixel, you measure the contrast among all of the primary rays. Pick the pixel(s) with the highest contrast, and shoot more primary rays in those pixels. You can also use the contrast between samples to choose where in the pixel to shoot new primary rays, but then you have to use a more complex reconstruction algorithm to avoid bias in your result.

If you can find a frame-grab of the particular details you noticed, it would be possible to go discuss those in more detail. It's entirely possible that some artefacts are caused by a particular technique, and aren't related to super-sampling at all.


Changing colors after raytracing Intro
Maxwell Render has a feature called multilight, where one can ray-trace a scene with multiple light sources, and change the brightness of the individual light sources afterwards.
Luxrender has a similar feature called light groups.
I image that behind the scenes, these renderers keep track of the contributions of the different light sources separately, and then composing a final image (given brightnesses of light sources) just amounts to mixing these per-light contributions with appropriate coefficients. Effectively, instead of keeping track of the color of a ray as a tuple (R,G,B), one keeps track of more numbers (R1, G1, B1, R2, G2, B2, ..., Rk, Gk, Bk) if there are k light sources (let's assume no spectral rendering, just for ease of notation -- it's conceptually identical).
From a mathematical point of view, the final image is a linear combination of the images resulting from the individual light sources.

Question
Now it seems interesting to have a similar feature that allows changing the colors of objects in the scene after rendering.
This can be done with an entirely similar technique: suppose that we have two objects whose color we want to be able to change after rendering. The final image is a polynomial in the color of these objects, with two variables (one for each object) and of degree n, where n is the maximum number of bounces for a ray.
Specifically, instead of keeping track of tuples (R,G,B), the integrator would keep track of
expressions of the form (R00,G00,B00) + (R10,G10,B10)x + (R01,G01,B01)y + (R11,G11,B11)xy + (R21,G21,B21)x²y + ..., for a total of k^n terms, where k is the number of colors you want to be able to change, and n the maximum number of bounces.
For modest k and n, this certainly seems tractable. Color calculations become more expensive, as does storing the resulting "polynomial-image", but all the geometric calculations are identical.

Is this feature implemented in any widely available renderers?

If so, which ones?

If not, why not? Is this not something that's useful in practice? Is it not computationally feasible?

Remark
I hope I've made myself clear in my explanation. I don't know much about computer graphics (mathematician by training), so my notation may be unusual. More general that just changing the colors of objects, one could change more properties of materials after rendering. For example, the transparency of objects could be changed afterwards (though assuming importance sampling is used by the integrator, this comes at the cost of noise, I guess).

What you describe is essentially a deep raster. Most modern renderers implement this in fact it is implemented in several ways. There are even sopecific technical positions that rely on this technology in form of lighting, color grading and compositioning artists

The first technique is multipass rendering which is basically just this dumping of render data out of the renderer for fast defered rendering.  All rendering engines on the planet can do this, though all do not implement dumping it out of one render, most do however). What you then do is you integrate these dumped buffers in a compositioning app (Nuke, Fusion, maybe even after effects, you can even do this in photoshop) where you can do final tweaks and changes. This is a very old idea and has been used for more than 30 years. It has certain drawbacks bit overall this means you night be able to pull off even calculation changes that would be too costly to do with a re-render. So color of objects

Many systems also support interactive sessions that cache the computations, in what is essentially a deep raster. This way its possible to recalculate not only color but even the position, decay of individual lights at interactive speeds. The most modern ones of these are frankly quite sophisticated and can also issue the renderer to rerender in background. Animation studios have lots of people working on these to make them better.

You can even use this trick to change camera position for the second stereo camera without recomputing your scene.


HLSL Deferred Rendering I've recently made the switch to Visual 2015 and the .fx files have been depreciated. My project contained .fx files for the shaders so I've been switching over.

Only during the switch have I changed to using deferred rendering for lighting.

The only shader I've made so far is the Pixel Shader that is used on the 2nd Pass, the shader that does the Per Pixel x Per Light calculations, sums them, and outputs final color.

However, whenever I compile (I'm not actually using the shader in runtime, I'm just getting it all setup, the window is still just cleared to a color every frame, no actual rendering) I get a failure that says....

error X4502: Shader model ps_4_0_level_9_3 doesn't allow reading from position semantics.

Here is the only shader I have in the project that is causing this error.

I omitted the getGBufferAttributes and calculateLighting methods because I know they are not the issue.

Texture2D NormalTexture                 : register(t0);
Texture2D DiffuseAlbedoTexture          : register(t1);
Texture2D SpecularAlbedoTexture         : register(t2);
Texture2D PositionTexture               : register(t3);

cbuffer LightParams
{
    float3 LightPos;
    float3 LightColor;
    float3 LightDirection;
    float2 SpotlightAngles;
    float4 LightRange;
};

cbuffer CameraParams
{
    float3 CameraPos;
};

float4 main(in float4 screenPos : SV_Position) : SV_Target0
{
    float3 normal;
    float3 position;
    float3 diffuseAlbedo;
    float3 specularAlbedo;
    float specularPower;

    getGBufferAttributes(screenPos.xy, normal, position, diffuseAlbedo, specularAlbedo, specularPower);

    float3 lighting = calculateLighting(normal, position, diffuseAlbedo, specularAlbedo, specularPower);

    return float4(lighting, 1.0f);
}

The code used for this shader is highly based off the code used as the deferred rendering example in the book, Practical Rendering with Direct3D 11 by Jason Zink, Matt Pettineo, and Jack Hoxley.

I don't understand why they would give source code that doesn't work. Does it have something to do with the new compiler in Visual 2015 and the Runtime?

Also, since I'm going from only using effects before to now raw HLSL shaders, is there anything I should be aware of that might trip somebody like me up?

A very novice mistake, I was compiling with the old HLSL compiler. But why would that be the default? Very strange.

For anybody wondering, just right click the HLSL file in the solution explorer and go to properties -> HLSL Compiler -> General, and switch Shader Model to the one you want which was newest one for me (5.0)


Need help with change of basis (world to camera frame) I'm reading up on coordinate systems and geometric transformations but the change of basis topic still confuses me. I found this exercise but, honestly, I wouldn't know how to begin:

  Given a world coordinate system in which the camera is at $(3,3,0)$ and pointed at the origin $(0,0,0)$ write an orthonormal set $(e_1, e_2, e_3)$ with an axis oriented like the camera axis and the matrix for the change of coordinates from world frame to camera frame $(O, e_1, e_2, e_3)$.

Do I need to use homogeneous coordinates or not? Currently, I'm using the book Interactive Computer Graphics: A Top-Down Approach with Shader-Based OpenGL. If anyone could suggest a better alternative I'd be grateful.

Since the transformation that is asked for includes a translation (the camera is located at a position other than the origin), you will indeed have to use homogeneous coordinates to describe this transformation in matrix form.

There are basically two ways to get to the matrix you're looking for. You can either set up the matrix that would transform from camera coordinates to world-space and then take the inverse of that, or you can build the matrix directly. The first approach is what is typically done and what the exercise seems to be aiming for since it explicitly asks for the base vectors of the camera system as seen from world-space, which is the first step in that process. The advantage of this approach is that it is arguably a bit more intuitive since what you're essentially doing is treat the camera as just another object that you place in world-space by setting up a matrix. Just that once you have that matrix, you invert it because you didn't actually want to get from camera-local coordinates to world coordinates but rather the other way around. I'll explain both approaches below using the concrete example given in your exercise. At least for now, I'll leave the generalization of the ideas presented below as an exercise to the reader (so that at least some of the fun remains for you ).

Let's look at a sketch of the whole situation:

$$
\def\mvec#1{\begin{pmatrix}#1\end{pmatrix}}
\def\ivec#1{\left(\begin{smallmatrix}#1\end{smallmatrix}\right)}
\def\mmat#1{\begin{bmatrix}#1\end{bmatrix}}
\def\vec#1{\mathrm{\mathbf{#1}}}
\def\mat#1{\mathrm{\mathbf{#1}}}
$$

Your camera is located at $\vec o = \mvec{3 & 3 & 0}^T$ and is supposed to be looking at the origin. By general convention, the camera is set up to be looking along the line of the z-axis of its local coordinate frame, while the local x-axis points right and the local y-axis points up. Therefore, we know that $\vec e_3$ should be pointing either at or away from the origin. Since you're talking about OpenGL, I assumed right-handed world and camera coordinate systems as is typically done in OpenGL-land. To get a right-handed coordinate frame where the local x points to the right, local y points up, and local z lines up with the viewing direction, $\vec e_3$ can only to be pointing in the direction away from the point at which the camera is looking. Once we have fixed $\vec e_3$, we have to define the remaining two base vectors. Note that no matter what we choose for $\vec e_1$ and $\vec e_2$, our camera will be looking at the origin. Our choice of $\vec e_1$ and $\vec e_2$ basically just fixes the rotation of the camera around the viewing direction, which is set by $\vec e_3$. Generally, one choses $\vec e_1$ such that it is orthogonal to both, $\vec e_3$, and a general "up" direction. $\vec e_2$ is then simply computed to be orthogonal to both, $\vec e_1$ and $\vec e_3$.

Approach 1

From the above sketch, it is immediately apparent that the coordinates of $\vec e_3$ have to be
\begin{equation}
  \vec e_3 = \frac{1}{\sqrt{3^2 + 3^2 + 0}} \mvec{3 \\ 3 \\ 0} = \frac{1}{\sqrt{2\cdot3^2}} \mvec{3 \\ 3 \\ 0} = \mvec{\frac{1}{\sqrt2} \\ \frac{1}{\sqrt2} \\ 0}.
\end{equation}
Since our camera is located in the x-y-plane in this example, the base vector $\vec e_2$ is just $\vec e_3$ rotated by $90°$:
\begin{equation}
  \vec e_2 = \mvec{-\frac{1}{\sqrt2} \\ \frac{1}{\sqrt2} \\ 0},
\end{equation}
and $\vec e_1$ just points in the direction opposite of the world-space z-axis:
\begin{equation}
  \vec e_1 = \mvec{0 \\ 0 \\ -1}.
\end{equation}
Using the world-space base vectors of the camera frame, we can write down a matrix that transforms from camera-space to world-space. The camera matrix that transforms from camera to world coordinates is then simply the inverse of that matrix:
\begin{equation}
  \mat M_{cam}^{-1} = \mmat{\vec e_1 & \vec e_2 & \vec e_3 & \vec o \\ 0 & 0 & 0 & 1}
         = \mmat{
    0 & -\frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 3 \\
    0 & \frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 3 \\
    -1 & 0 & 0 & 0 \\
    0 & 0 & 0 & 1}.
\end{equation}
Since the camera frame has an orthonormal base just offset by $\vec o$, this matrix is composed of just the rotation defined by the orthonormal base vectors followed by a translation:
\begin{equation}
  \mat M_{cam}^{-1} = \mat T \mat R
\end{equation}
where
\begin{align}
  \mat R &= \mmat{
    0 & -\frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 \\
    0 & \frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 \\
    -1 & 0 & 0 & 0 \\
    0 & 0 & 0 & 1} &
  \mat T &= \mmat{
    1 & 0 & 0 & 3 \\
    0 & 1 & 0 & 3 \\
    0 & 0 & 1 & 0 \\
    0 & 0 & 0 & 1}.
\end{align}
You can convince yourself that this is true by simply performing the multiplication. Knowing this, we can compute the camera matrix:
\begin{equation}
  \mat M_{cam} = (\mat T \mat R)^{-1} = \mat R^{-1} \mat T^{-1}.
\end{equation}
Since $\mat R$ is composed of orthonormal base vectors, it is an orthogonal matrix, which means that its inverse is just its transpose. $\mat T$ is just a translation, its inverse is a translation in the opposite direction. Therefore,
\begin{equation}
  \begin{split}
  \mat M_{cam} &= \mmat{
    0 & 0 & -1 & 0 \\
    -\frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 & 0 \\
    \frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 & 0 \\
    0 & 0 & 0 & 1} \cdot \mmat{
    1 & 0 & 0 & -3 \\
    0 & 1 & 0 & -3 \\
    0 & 0 & 1 & 0 \\
    0 & 0 & 0 & 1} \\
    &= \mmat{
    0 & 0 & -1 & 0 \\
    -\frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 & 0 \\
    \frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 & -3\sqrt2 \\
    0 & 0 & 0 & 1}.
  \end{split}
\end{equation}

Approach 2

An alternative to the approach above is to just directly compute the coordinates of the world-space base vectors as seen from the perspective of the camera. Given these coordinates, we can, again, just write down a matrix that applies the corresponding transformation from world to camera space.

Looking at the figure again, we see that the coordinates of the world-space x-axis as seen from camera space are
\begin{equation}
  \vec x' = \mvec{0 \\ -\frac{1}{\sqrt2} \\ \frac{1}{\sqrt2}}.
\end{equation}
Analogously, we find the camera-space coordinates of the world-space y- and z-axes:
\begin{align}
  \vec y' &= \mvec{0 \\ \frac{1}{\sqrt2} \\ \frac{1}{\sqrt2}} &
  \vec z' &= \mvec{-1 \\ 0 \\ 0}.
\end{align}
Also looking at the figure, we note that the world-space origin as seen from the camera is simply located at a distance of $\sqrt{3^2 + 3^2} = 3\sqrt{2}$ units along the negative $\vec e_3$ axis. Therefore, the camera-space coordinates of the world origin are
\begin{equation}
  \vec o' = \mvec{0 \\ 0 \\ -3\sqrt{2}}.
\end{equation} and we can, once again, write down the matrix
\begin{equation}
  \begin{split}
  \mat M_{cam} &= \mmat{
    0 & 0 & -1 & 0 \\
    -\frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 & 0 \\
    \frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 & -3\sqrt2 \\
    0 & 0 & 0 & 1}
  \end{split},
\end{equation}
and sleep well tonight because both ways did indeed lead to the same result…


Flat shading for non-planar polygons I'm trying to figure out how to properly implement flat shading for meshes containing non-planar polygons (using OpenGL/GLSL). The aim is to obtain something similar to the result Blender gives (all polygons below are non-planar):

To provide a bit of context — the non-planar polygons (e.g. quads, pentagons, ...) are either present in the mesh I load or appear as a result of applying a subdivision scheme (such as Catmull-Clark) to the mesh.

In case of a triangle mesh, I'm aware of the following approaches to implement flat shading —

Compute the cross product of dFdx and dFdy in the fragment shader
Use the geometry shader to compute the triangle normal by taking the cross product of two sides of the triangle
When using glDrawElements, use the flat interpolation qualifier (which basically uses the normal associated with the provoking vertex for all fragments)
When using glDrawArrays, most vertices (usually all of them) are copied to the GPU multiple times. Therefore, a different normal can be used each time

For a mesh containing non-planar polygons the first three options don't yield the result I'm after. In my implementation (using the Qt framework) I use GL_TRIANGLE_FAN:

The fourth option could work, but isn't very elegant — I'd like to use either glDrawElements or glDrawMultiElements. One other option would be to use glDrawElements while looping over the faces one by one (instead of invoking it once using glPrimitiveRestartIndex). In that case, one could upload a uniform normal for each face. However, this does not seem very efficient. Any other ideas?

Without knowing the internals of Blender, I would say it uses shared normals for the quads, splitting the normals only on edges between the quads, not between the triangles. So your idea of uploading a normal per quad is probably close to the result you are looking for.

If you load the mesh as quads, you are set. If you are using an old version of OpenGL where quads still exist, you may even be able to use solution 3).

If you load the mesh as triangles, you will have to figure out which triangles originated in quads. You can likely do this simply by enforcing split normals beyond a fixed angle between adjacent triangles.

I am not sure there is a simple dynamic solution, where you just render any mesh. You may be able to use the geometry shader to look at the adjacent vertices, but the easiest version is to smooth the normals once before upload.


Calculate the position and rotation needed to have a camera look at a certain region I have a 2D region defined by 4 points in 3D space. I want to position my camera so that it looks at the region, with a settable variable determining which side it looks at, where looks at means that the camera view covers the entire region to be looked at, and as little beyond that as possible. How do I calculate the orientation and position of the camera?

I have a 2D region defined by 4 points in 3D space. I want to position my camera so that it looks at the region, with a settable variable determining which side it looks at,...

The first thing you need to do for this is to define a convention that specifies which side is the front and which one is the back of your plane depending on the ordering of your nodes. With triangles, you usually use the winding order. So you can define that if you look at the front of a triangle, the points have to be in clockwise order. In conclusion, if you see that the nodes of the triangle are ordered counter-clockwise, you look at the back.

  How do I calculate the orientation and position of the camera?

Once you have this convention, you have to calculate the normal of the surface. You can get it by calculating the cross product of 2 vectors. For example, if your points are named A, B, C, D, take the vectors A->B and A->C and calculate their cross product to get the surface normal. Be aware, that the cross product is sensitive to the vector/node order. So make sure the selected points result in a normal that points away from your front side.

Now, your camera orientation is the negative normal direction if you look at the front and the normal's direction if you look at the back.

Then you calculate the center point of your quadriliteral. There are several ways to do this, so just pick your favorite method or search the internet for it. Your camera position C has to be on the line that has the same direction as your surfaces' unit normal N and goes through the center point P of your surface. So it can be calculated by

$$
C = P + d \cdot N
$$

where d is the distance of the camera to point P. The distance depends on your camera's field of view and the size of your plane. In your special case, where the camera looks directly onto your surface, the camera/screen plane is parallel to your surface. So you can use the intercept theorem to calculate the minimal necessary camera distance so that all points of your plane are inside the cameras FOV.

Remark:

The described method leaves you with one degree of freedom for your camera that is chosen more or less randomly. The rotation angle around its view direction. Depending on the size and shape of your plane, the rotation angle also influences the camera's distance to the plane. So you might want to define an additional constraint to always get the same result. For example, you could say, that your surfaces side A-B has to be parallel to your screens left-hand side.


is utilizing multiple GPUs better than using one for all software? In cases of Laptops, some will have two GPUs. One is built-in (e.g. Intel) and another one attachable (e.g. Nvidia).
Is it better to have each of them operate different software to divide the load?
or
is it better to have one of them operate everything because the switching between GPUs might cause slower performance?

This is more of a long comment than an answer.
As with most yes or no questions in graphics, the answer is "it depends on the task". However, one particular task that you might find having multiple gpu's useful is debugging. For example if you are using cuda-gdb for debugging your CUDA application, it would be nice to have multiple gpu's, one for your application, the other for the display manager of your system. Note that cuda-gdb also works on a single gpu as well, but I had problems with it when I run the debugger and the display manager of the system on a single gpu (screen tends to freeze, no response from the system, etc).

Update 19-01-2021 with respect to comment.
Here is the main criteria that I would use for deciding if the use of single or multiple gpus are better for the task.
GPUs simply execute one task after another, so if you send a heavy task, it will do only that during the process and you won't be able to do anything else with it.
This usually means that your system would freeze until the gpu is done with your task.
If you are going to launch the task and won't plan to do anything with the machine until it is finished, then sure, the use of multiple gpus would get it done quicker. If you plan to use the machine during the process for other stuff, then it would be more reasonable to allocate a single gpu for the task.
If you are setting up some kind of rendering server where the server would do only gpu related computation, it depends on the type of request you are planning to process by the server.


Gamma setting in games - Why 3 icons? Whenever I try to adjust gamma settings in a game, I am shown three different icons and usually asked to make sure the left-most one is barely visible.

Example:

What are the other two icons for?

This has to do with gamma correction.

If one pixel has red component with value of 1 (where 255 is max), the next pixel has value for red that is 2, there no guarantee that exactly twice as much photons are exiting from the second pixel. Displays have different curves that predict what's the expected brightness. It also has to do with how our eyes work: they're more sensitive for darker shades than brighter shades.

To read more about these phenomena:

https://gamedevdaily.io/the-srgb-learning-curve-773b7f68cf7a#.di88vr58t
http://marcinignac.com/blog/pragmatic-pbr-setup-and-gamma/ (gamma section)
http://colour-science.org/posts/the-importance-of-terminology-and-srgb-uncertainty/
http://www.essentialmath.com/GDC2015/VanVerth_Jim_DoingMathwRGB.pdf

The three images are used as low value, medium grey and bright examples. Indeed, the lowest brightness is used so you are able do see the difference between the black background and "barely lit" object, and the other two images are just used to display other important values for the set gamma curve.


Can a scene graph be stored in the GPU? I am starting to learn WebGL. Can a scene graph be stored in the GPU? Or is that type of thing more for the main thread (or webworker)?

For example, I want to calculate some layout, and apply it as a transform matrix. Can I keep track of a scene graph node in the GPU over time, and modify it's properties over time?

I'm just wondering because I know the GPU architecture is different. Do people normally do scene graph stuff outside the GPU? Can it be done in the GPU?

Short answer: Yes, It can be done. But no one does so.

Long answer:
Scene graphs can be stored and processed on a GPU using OpenCL/WebCL. But it is not practical to do so. Updating scene graphs (a tree not in flat memory) on a GPU is slow, and processing it on a GPU is also slow (again, the tree is not in flat memory), while computing transformation matrices is cheap on a CPU. Thus no one is doing this.


How to set flags and usage glnamedBufferStorage in new OpenGL 4.4+ (GL_STATIC_DRAW)? In the documentation I see that the final parameter for glNamedBufferStorage is flags and that flags:

  Specifies the intended usage of the buffer's data store. Must be a bitwise combination of the following flags. GL_DYNAMIC_STORAGE_BIT, GL_MAP_READ_BIT GL_MAP_WRITE_BIT, GL_MAP_PERSISTENT_BIT, GL_MAP_COHERENT_BIT, and GL_CLIENT_STORAGE_BIT.

However, in the older glBufferData we use to specify a usage flag which would include things such as GL_STATIC_DRAW to specify expected usage patterns.

Where do we include that with the newer OpenGL?

The usage flag in BufferData was basically useless. There is no functional difference between each value and drivers used other ways to decide how to handle its data.

The flags parameter on the other hand actually changes what you can do with the buffer.


Sending shadow rays to the sun How is sun modelled for ray tracing rendering? Is it part of the scene, like a
sphere with a centre in some far location or it is just considered as a point?

Also regarding sending shadow rays to the sun, is it, again, sampling a sphere
or just sending shadow rays to one point?

Sun light can be represented as a distant light. Light sources are radial like a bulb, though in the case of the sun, it is so far away from the Earth, that a reasonable approximation is to consider that all rays coming out of the sun are parallel at the Earth's surface. Using a distant light is an approximation. Because it's far away it is also assumed that there is no variation of light intensity as you go up or down (change height).
If you do physically based rendering, the sun is an area light. From the Earth it looks like a small disk so you can approximate this with a large disk that is far far away. Area light like all light sources obeys a light intensity square fall off. So you will need to put your area light far away so that the intensity on the ground or as you change height doesn't vary much. Representing the sun as an area light avoids having to do strange things to the surface shader such as specifying a specular size.


How to read vertex color before rendering texture? I'm using VBO to upload vertex array, colors and then texture coordinates! I did implement mouse pick with color coding technique. It works fine but when I render texture after ReadPixels, it returns wrong color!
I don't know why this happens! Here is my render code:

        GL.PushClientAttrib(ClientAttribMask.ClientVertexArrayBit);
        GL.ShadeModel(ShadingModel.Flat);
        Debug.Assert(_vbo.VertexBufferId != 0);
        Debug.Assert(_vbo.ElementBufferId != 0);

        // vertex colors
        if (_vbo.ColorBufferId != 0)
        {
            GL.BindBuffer(BufferTarget.ArrayBuffer, _vbo.ColorBufferId);
            GL.ColorPointer(4, ColorPointerType.UnsignedByte, sizeof(int), IntPtr.Zero);
            GL.EnableClientState(ArrayCap.ColorArray);
        }

        // vertex position
        GL.BindBuffer(BufferTarget.ArrayBuffer, _vbo.VertexBufferId);
        GL.VertexPointer(3, VertexPointerType.Float, Vector3.SizeInBytes, IntPtr.Zero);
        GL.EnableClientState(ArrayCap.VertexArray);

        // primitives
        GL.BindBuffer(BufferTarget.ElementArrayBuffer, _vbo.ElementBufferId);
        GL.DrawElements(OpenTK.Graphics.OpenGL.PrimitiveType.Triangles, _vbo.NumIndices * 3, _vbo.Is32BitIndices ? DrawElementsType.UnsignedInt : DrawElementsType.UnsignedShort, IntPtr.Zero);

        // Read pixel under mouse cursor
        _selectedPolyIndex = uint.MaxValue;
        Byte4 Pixel = new Byte4();
        GL.ReadPixels(mouse_x, height - mouse_y, 1, 1, PixelFormat.Rgba, PixelType.UnsignedByte, ref Pixel);
        _selectedPolyIndex = Pixel.ToUInt();

        // Restore the state
        GL.PopClientAttrib();

        GL.PushClientAttrib(ClientAttribMask.ClientVertexArrayBit);

        // normals
        if (flags.HasFlag(RenderFlags.Shaded))
        {
            if (_vbo.NormalBufferId != 0)
            {
                GL.BindBuffer(BufferTarget.ArrayBuffer, _vbo.NormalBufferId);
                GL.NormalPointer(NormalPointerType.Float, Vector3.SizeInBytes, IntPtr.Zero);
                GL.EnableClientState(ArrayCap.NormalArray);
            }
        }

        // UV coordinates
        if (flags.HasFlag(RenderFlags.Textured))
        {
            GL.Clear(ClearBufferMask.ColorBufferBit | ClearBufferMask.DepthBufferBit); // this line must move out to line 160
            if (_vbo.TexCoordBufferId != 0)
            {
                GL.BindBuffer(BufferTarget.ArrayBuffer, _vbo.TexCoordBufferId);
                GL.TexCoordPointer(2, TexCoordPointerType.Float, 8, IntPtr.Zero);
                GL.EnableClientState(ArrayCap.TextureCoordArray);
            }
        }

        // vertex position
        GL.BindBuffer(BufferTarget.ArrayBuffer, _vbo.VertexBufferId);
        GL.VertexPointer(3, VertexPointerType.Float, Vector3.SizeInBytes, IntPtr.Zero);
        GL.EnableClientState(ArrayCap.VertexArray);

        // primitives
        GL.BindBuffer(BufferTarget.ElementArrayBuffer, _vbo.ElementBufferId);
        GL.DrawElements(OpenTK.Graphics.OpenGL.PrimitiveType.Triangles, _vbo.NumIndices * 3, _vbo.Is32BitIndices ? DrawElementsType.UnsignedInt : DrawElementsType.UnsignedShort, IntPtr.Zero);

        // Restore the state
        GL.PopClientAttrib();

When texture rendering is enabled OpenGL reads wrong pixel color under mouse cursor (I want vertex color but it reads texture color), even I call readpixels before texture rendering!

How can I fix this problem?

For this kind of functionality, you shouldn't try to use the same framebuffer for the vertex colours and for the final image. Don't forget that GL is asynchronous, so alternating CPU reads and GPU writes to the same framebuffer seriously limit your achievable performance and make it much harder for you to get the concurrency right. (The OpenGL wiki has more about synchronization.)

Instead, you should create an FBO that you only use for your vertex colours render. Draw the vertex colours into that and use it for mouse picking. Only use the 'real' framebuffer (the one that gets drawn) for the final image.


vertical or horizontal line test I have a line segment defined by two coordinate values and I need to check if the line segment is parallel to the x-axis or y-axis. I believe that it is not that difficult to find. When it comes to test for parallelism against x-axis, we just need to check if the both the end points of the line segments has the equivalent y-value, whereas parallelism test against y-axis can be found if both the coordinate point ends have the same x-value.

Any other better way do it is most welcome.

Thanks

It is not generally useful to try to optimise small parts of a program before you have profiled the program to see where the most benefit can be gained by optimisation. Make sure everything is correct and working first, and then profile to decide where to optimise (if optimisation turns out to be necessary).

Mathematically, your approach is already correct for all lines. It will correctly identify all lines parallel to either axis, and it will not incorrectly identify any lines that are not parallel to an axis. However, for use in a practical setting, it's worth asking yourself about near-parallel cases and any odd exceptions. Consider all this before even thinking about optimisation, otherwise you may have to repeat all the optimisation later if something you have to change for correctness breaks your initial optimisation.

One case to consider is when both the coordinates are identical. That is, the end points have the same x coordinate and the same y coordinate. This is a  degenerate case that doesn't define a line (it could have any direction). Check your implementation with this case and make sure it gives the output that you want it to.

Depending on your purpose, you may want to detect only exactly parallel lines, or you may want to detect lines that are sufficiently near to parallel. In some cases lines near to parallel may satisfy your purpose, and in other cases it may be essential to accept lines near to parallel, if for example there is the possibility of small inaccuracies in the coordinates.


Using GPU in PC for GearVR Usually GearVR + SamsungPhone are used both for rendering and displaying but I would like to use it only for displaying when it is plugged to PC. The reason is to get performance similar to Oculus Rift or HTC Vive and not spend money for it. Is it possible?

Since you're asking in this Stack Exchange, I assume you intend to build a VR app yourself that utilizes this scenario.

While it is perfectly possible to utilize PC for rendering and stream the output to a mobile VR headset, it will come at the relatively high cost of increased latency (see e.g. here for why this is important; the tl;dr is that high latency → nausea). This is because wireless networks have intrinsically higher latency than wired ones, and you need to do the following in a loop:

Collect head tracking (gyroscope/accelerometer) data from the VR headset.
Send the head tracking data over the network to the PC.
Render a frame on the PC.
Stream the frame from the PC to the VR headset. This involves the following substeps:

Encode video and audio on the PC.
Transmit the encoded media frame to the VR headset over the network.
Decode video and audio on the VR headset.

As I mentioned previously, steps 2 and 4 introduce additional latency (probably from several to a dozen milliseconds, from stream encoding/decoding and network transmission), which – depending on your specific application and the individual tolerance of the user – may or may not be a deal breaker.


How to properly direct light sample GGX in path tracing I have been trying to implement the GGX shader in my path tracer based on this paper: Microfacet Models for Refraction through Rough Surfaces
It's pretty clear how the importance sampling works, but I would also like to do next event estimation (direct lighting) for when the object is more rough and acts close to a diffuse shader.

The part that is really confusing to me is equation [19] and [20]:

I don't understand why in [19] they use the microsurface normal, but in [20] they use the macrosurface normal. The transmission term I don't need because I'm just working on the reflectance, but I still don't get why it's m on [19] and n on [20]. Is it possible to use any incident direction and outgoing direction and get back a proper BRDF? Or does the scatter direction "o" have to match where the microsurface normal makes it bounce to? In direct lighting the directions would need to be independent of each other for it to work since you're grabbing a from the hit point to the light, and from the hit point to the camera.

This may simply be a typo or oversight. Note that in the previous section they've been discussing the microsurface BxDFs $f^m_r(\mathbf{i},\mathbf{o},\mathbf{m})$ and $f^m_t(\mathbf{i},\mathbf{o},\mathbf{m})$, which are naturally evaluated at a microsurface normal, but in this section they've moved on to talking about macrosurface BxDFs, which have the microsurface integrated out.

AFAICT, the intent of equation 19 is just to say that the total BSDF is a sum of reflection and transmission parts, so it doesn't really matter ($\mathbf{m}$ is a free variable there and could be named anything), but to be less confusing it should probably have been $\mathbf{n}$. Maybe they just forgot to update it during a round of edits, or something.

Yes, you can evaluate a BRDF/BSDF with any combination of input and output directions and it should give the correct answer.


Why is the line from the camera to vanishing point parallel to the other parallel lines? When trying to find the vanishing point of an image, why is it that the vector from the center of the camera to the vanishing point is parallel to the parallel lines of the image?

What's the mathematical reasoning behind this?

FYI every set of parallel lines will have their own vanishing point. The reason why there is usually only a single vanishing point (the most obvious one) talked about in an image is due to aesthetic choice.

A perspective projection can be seen as a point projection of a 3D scene onto a 2D plane. The way to determine where a point in the 3D scene ends up on the 2D plane is to draw a line between the point and the camera (the sight line) and see where it intersects the projection target plane.

To find the vanishing point of a set of parallel lines you can take a the limit of traveling on that line infinitely far. At the limit the sight line will be parallel to the line.


Do I need a normal matrix if I calculate lighting in an objects space? As a quick reminder, here is what normal matrix is and why do we need it.

Now the question - if I take the light position and transform it into an object space than I don't need the normal matrix at all and actually it will save me a good amount of calculations (don't need to multiply a normal by the matrix). By I can't draw a mental picture of how this could work for a scaled object, what does it mean to transform a point (light position) into a scaled object space? And what about a directional light where I have no position at all?

By lighting in object space, sure, you could avoid transforming the normals into world space. However, you'd have to transform the light positions/vectors into the object's space. Also, if you want any view-dependent shading such as specular, you would have to transform the camera position into object space as well, to calculate view vectors.

Now, all of this can be done straightforwardly using the inverse of the object's world matrix. The inverse world matrix will include the inverse of the object's scale (as well as correctly accounting for the inverse of its rotation and translation), so it will transform the light & camera positions to the appropriate points in the scaled coordinate system. Directional lights don't have a position, but you can transform their direction vector by the inverse world matrix to account for rotation and scale. All of this is fine so far.

However, you will then run into trouble when doing lighting calculations like N·L or N·V. For exactly the same reason that normals don't transform correctly under non-uniform scaling without using a special normal matrix, these dot products will not come out with the same result when done in a non-uniformly scaled object space as compared to world space. Namely, non-uniform scaling distorts angles. So, all your lighting calculations and BRDFs that depend on angles are going to be messed up.

While there are ways one could correct this problem by using additional matrices when calculating dot products... it's generally easier and faster to do lighting in world space (or view space). In any case, transforming the normals to world space is a pretty trivial amount of math by modern GPU standards, especially as it only needs to be done per vertex.


Technology for specific project 3d in c#? I want to write a program that allows me to simulate the effect of wind on a decorated 3d tree standing outdoors (rocking ornaments and decorations, falling balls, etc.). The program should allow to determine the parameters of the tree, the selection and placement of ornaments and the specification of wind parameters (wind way, average force, gusts, etc.) Does anyone know what libraries or tools I could use I have heard about unity, does anyone have any suggestions?

I have done 3D graphics in c# with Managed DirectX. Nowadays people uses SharpDX and SlimDX instead. I have heard about XNA is easier for doing simple 3D games.

You can also use OpenGl since there are several bindings for C#. Some people find OpenGl simpler to understand than DirectX, and is cross platform.

Unity uses C# as scripting language, the Unity Engine is running the actual simulations. You are mostly configuring and scripting Unity for running your simulation. This is not just a technology to use in your C# project, this is the oposite, a platform for games that uses C# as one technology.

You can do 2D graphics with all the above technologies. For the simulation, in case you do not use Unity or other game engine capable of doing physics simulations, you would need to find or write a rigid body simulation for animating all your moving objects. For animating the tree you would probably need a spring system which is also simple to simulate along rigid bodies.


What is the basis of rendering light caustics? I am not looking for textbooks and related how-tos, though I'm sure they can color the answer.

I'd like to understand how we handle the rendering of caustic effects — and particularly time-saving models, as different from physical simulation.

For offline rendering, caustics and dispersion effects (different colors of light separating, as in a prism) are usually calculated using some form of photon mapping. That's a fairly expensive technique, even by the standards of offline rendering, and it falls more under the "physical simulation" side of things.

If you're looking for more approximate techniques suitable for real-time rendering, Chris Wyman's papers are a good place to look. He wrote several papers on caustics between 2006 and 2009. The one to start with is Interactive Image-Space Techniques for Approximating Caustics (I3D 2006), #25 on that page. This paper re-uses a GPU refraction rendering technique he developed earlier, #28 on that page, so you'll want to look at that as well for background. Later papers build on these two.

The basic idea of these papers is to use GPU rasterization to perform a limited, approximate analogue of photon mapping. The scene is rendered from the light's point of view, similar to a shadow map, and each pixel is treated as a photon; its refraction through transparent objects is calculated approximately (usually considering only 1 or 2 layers of refraction), and its final location is stored in a texture. Then, one can render each photon as a small splat into a "caustic map", which finally gets projected onto the scene, similar to a shadow map.


realtime dynamic reflections, which way to go? I'm working on my own engine and I'm struggling to find any techniques that give okay-ish reflections in realtime on dynamic scene.
So far the only technique I found that work in real time on dynamic scene is SSR but it's really heavy and have a lot of bias.

The idea of having multiple cubemaps and updating them in real time crossed my mind but because that mean re-rasterising the scene that also mean that every cubemap updated in realtime is a shadow casting that I can't do.

So my question is: is there another technique other than reflection probe (cubemap) or Screen Space Reflections ?

The artstyle of the game where the engine will be used is based on a lot of "almost mirror" reflections and it's not PBR.

[Edit]
To be a bit more precise with what I'm searching:

I do not need more than one bounce.
I do not want any baking that cannot be done at "loading time".
I prefer if the technique work with a per-pixel normal but if you
know a technique that work with a per-vertex normal I am also open.
I am open to techniques that only work in specific cases (like tiny
scene, deferred rendering only etc...) as long as it meet other
requirements.
If your technique is a feature of a global illumination technique I am
also open but I would prefer if it work stand alone.
If it can easily support GGX it's better but not required.
Inaccuracy is not a problem as long as it is predictable by the
artist

If there is only a single flat mirrored surface the solution is to render the scene twices, once normally. Afterwards you use stencil to mark the area where the mirror is visible.

Then only drawing into that area you render the scene again but mirror it around the mirror's plane and cull any geometry on the other side of the mirror.


How do people come up with subdivision schemes? Be it chaikin subdivision, loop subdivision, catmull-clark subdivision...

How do people come up with the coefficients for an arbitrary subdivision scheme?

The subdivision schemes are not arbitrary. Catmull-Clark, arguably the most used subdivision scheme, generalizes bicubic B-splines to meshes of arbitrary topology.

Most, other subdivision schemes also generalize other degree B-splines to arbitrary topology. Doo-Sabin for biquadratic B-splines and Loop subdivision generalizes quartic box-splines defined over regular triangulations.

Most of the coefficients for subdivision are obtained by looking at knot-insertion. Knot-insertion is the process of adding knots into B-splines such that the same curve is obtained by expressing it as more curve pieces, i.e. subdividing the original curves. The rules for subdivision and thus the coefficients are derived from these algorithms. However, these rules only hold for the regular regions of the control mesh. For the irregular regions the rules are generalized, such that they give a satisfactory result.

Another way to look at it is through subdivision stencils and masks. The simplest subdivision stencil $[1,1]$ applied to the control polygon with points $P_i, \ldots, P_N$ will generate new points $Q_i = P_i$, will simply insert points at the same positions as the original points.
We can convolve the stencil with itself $[1,1] * [1,1] = [1,2,1]$. We can decompose this stencil into an even [1,1] and an odd mask [2] and again apply it to the control polygon by multiplying the coefficients in the mask and dividing by the sum of coefficients. The odd mask $Q_{i*2} = 2 P_i / 2$ will keep the original points, but the even mask will generate points $Q_{i*2+1} = (1 P_i + 1 P_{i+1})/2$ half-way between points.

Continuing the convolution process to the second stencil $[1,2,1] * [1,1] = [1, 3, 3, 1]$ will generate odd $[1, 3]$ and even $[3,1]$ masks which will generate points one quarter and three quarters along the line piece between points $P_i$ and $P_{i+1}$. This are of course the exact coefficients used in Chaikin's corner cutting algorithm, which in the limit of subdivision will generate quadratic B-splines. Applying it again $[1,3, 3,1] * [1,1] = [1, 4, 6, 4, 1]$ will generate the stencils that generate cubic B-splines.

To generalize this again to surfaces we can apply this convolution process in two parametric directions to obtain bi-degree B-splines. In the cubic case we obtain this matrix of coefficients through convolution:

$$\begin{bmatrix}1 & 4 & 6 & 4 & 1\\
                4 & 16 & 24 & 16 & 4\\
                6 & 24 & 36 & 24 & 6\\
                4 & 16 & 24 & 16 & 4\\
                1 & 4 & 6 & 4 & 1\\ \end{bmatrix}$$

We can decompose this matrix into vertex mask:

$\begin{bmatrix}1 &  6 & 1\\
                6 & 36 & 6\\
                1 &  6 & 1\\ \end{bmatrix}$

edge point mask:

$\begin{bmatrix}4 &  24 & 4\\ 4 & 24 & 4\\ \end{bmatrix}$ and $\begin{bmatrix}4 & 4\\ 24 & 24\\ 4 & 4\\ \end{bmatrix}$

face point mask:

$\begin{bmatrix}
16 & 16 \\
16 & 16 \\
\end{bmatrix}$

Applying these as control point masks to a regular control grid will obtain bicubic B-splines. The observation made for the Catmull-Clark algorithm is that the inserted points through the face stencils are the average of the surrounding vertices. This can easily be generalized for arbitrary polygons, the same thing goes for the edge stencil. The hard case is the vertex stencil as the original is only valid for vertices with valency 4. Catmull-Clark came up with this expression

$$\frac{Q}{N} + \frac{2R}{N} + \frac{S(N-3)}{N}$$

Where $S$ is the old vertex position $R$ is the average of the points created by the edge mask for the outgoing edges of the vertex and $Q$ the average of the points created by face mask for the surrounding faces. The ratios in this expression are such that it reproduces the regular vertex mask in the case $N=4$ and is aesthetically pleasing for other $N$.


OpenGL - Colours not working properly I am using a single VBO to store vertices in the follow format:

v1X, v1Y, v1Z, v1R, v1G, v1B, v2A,
v2X, ...

Vertex positioning is fine, shapes show up where expected, however instead of using the colour provided, all shapes show up red.

The code given below simply draws two triangles to form one square ground shape.

Buffer data preparation method

public void prepare(float[] data) {
    glBindBuffer(GL_ARRAY_BUFFER, vboID);
    FloatBuffer dataBuffer = RenderUtils.fArrayToBuffer(data);
    if(dataLength != data.length) {
        glBufferData(GL_ARRAY_BUFFER, dataBuffer, GL_STATIC_DRAW);
        dataLength = data.length;
    } else {
        glBufferSubData(GL_ARRAY_BUFFER, 0, dataBuffer);
    }
    glVertexAttribPointer(0, 3, GL_FLOAT, false, 7*4, 0);
    glVertexAttribPointer(3, 4, GL_FLOAT, false, 7*4, 3*4);
}

Render code

floorObj.prepare(new float[]{
    -5, 0, -5,   1, 0, 0, 1,
    5, 0, -5,   0, 1, 0, 1,
    -5, 0, 5,   0, 0, 1, 1,

    5, 0, -5,   1, 0, 0, 1,
    5, 0, 5,   0, 1, 0, 1,
    -5, 0, 5,   0, 0, 1, 1,
});
glDrawArrays(GL_TRIANGLES, 0, 6);

Vertex shader

#version 330 core

in vec3 position;
in vec4 i_color;
out vec4 color;

uniform mat4 transform;

void main(){
    gl_Position = transform * vec4(position, 1);
    color = i_color;
}

Fragment shader

#version 330 core

in vec4 color;
out vec4 f_color;

void main(){
    f_color = color;
}

As previously stated, vertex positions work fine, however colour does not.
Just ask if any other code would be useful to determine the problem.

Thanks, - Jasper

Your attribute locations could be wrong, because you're not querying them:

glVertexAttribPointer(0, 3, GL_FLOAT, false, 7*4, 0);
glVertexAttribPointer(3, 4, GL_FLOAT, false, 7*4, 3*4);

The first parameter should be queried using glGetAttribLocation:

glVertexAttribPointer(glGetAttribLocation( program, "position" ), 3, GL_FLOAT, false, 7*4, 0);
glVertexAttribPointer(glGetAttribLocation( program, "i_color" ), 4, GL_FLOAT, false, 7*4, 3*4);

If using more modern OpenGL, the locations can be set manually in the shader, eliminating the need for glGetAttribLocation:

layout(location=0) in vec3 position;
layout(location=3) in vec4 i_color;


What are the usages of software rasterization in modern game engines? I am taking a Computer Graphics course this quarter.
One of our lab project is about software rasterization.

Now I am planing the project proposal and
thinking about how to make it useful for other people in contemporary game development.

After some brief search,
I learned a technique called Software Occlusion Culling.
It does software rasterization on buffers of various resolutions.
And we can query for occlusion using the hierarchical z buffers.

My question: What are the usages of software rasterization
in modern game engines besides the Software Occlusion Culling?

To my knowledge, Software Occlusion Culling (which you already mentioned), is pretty much the only thing a software rasterizer would still be used for. Procworld makes use of a similar technique to display its huge voxel environments.

Most other culling methods like frustum culling do work on the CPU, but, to stick with the example, the test against the frustum happens on object level, probably with an axis-aligned bounding box (AABB). This intersection test is way simpler than using a full-blown software rasterizer.

In certain cases, software rasterization could be used for mouse picking of objects. In game engines, this is often solved using a physics engine and ray-triangle collision with a simplified mesh.

With the CPU being idle while waiting for the GPU in modern interactive 3D applications, one could think that it might be beneficial to use these idle cycles to render on the CPU using a software rasterizer. The problem here, besides the rendering getting horribly complex and convoluted, will often be the bandwidth. Images rendered on the CPU need to be transferred to the GPU before getting displayed, which might cancel out the benefit.


How to project a 3D point onto a plane along another (axis) vector? I have a point $P = (0, y, z)$. I need to determine the $x$ coordinate of its projection $P'$ onto a plane $T$ along the $x$ axis unit vector $\vec{X} = (1, 0, 0)$. The plane $T$ is defined by a point $V$ that lies on $T$ and its normal $\vec{N}$ (see the attached picture).

You can determine x by calculating line-plane intersection. Your line starts at P and has direction D=[1, 0, 0], and plane T=[V, N]. This can be done as follows:
$$x=\frac{(V-P)\cdot N}{N\cdot D}$$
x is the distance from P along D to the intersection point with T (assuming both D and N are unit vectors)


Blackbody curve to floating RGB I want to set the color of the lamps in my game with a black body curve, but the approximations I can find for it are all in sRGB color space; and I want it in linear floating RGB.

The closest thing i was able to find is this: https://mathematica.stackexchange.com/questions/57389/convert-spectral-distribution-to-rgb-color

But i really don't understand most of the notation in the answer.

Mainly I'm unsure: if i just use planck's law, using the wavelengths of the subpixels in a standard monitor--not that I know what those are--will it get the right answer, or would it need to be an integral with respect to cone response.

Furthermore I'm not sure how to normalize the resulting radiation density from.... whatever units its in, to 1.0 = X nits; as I'm unsure of the conversion between the units...

The Open Shading Language has a complete implementation which is based on this source code from Color Rendering of Spectra by John Walker, according to the comments. I can't vouch for its theoretical correctness but I have used it and it appears to work reasonably well.


Model Separation - Several models reside in a single .stl file I have an .STL file (ascii and binary format) that contains several different CAD models. How can I read the file and create separate .stl files for each of the single different models. Even some hints/reference would be helpful.

Software recommendation are not on topic. Instead I will explain how you separate the shapes. Each object is a separate volume so you can just extract all the continuous shells. You can do this simply by picking a vertex marking it as visited and then picking all connected vertices and marking them visited. You continue this visiting to new un-visited vertices until you no longer can find a connected vertex.

This is a shell, in most cases a shell is a separate object. Unless there is a empty non connected cavity inside the object. This is rare but possible in 3d printing though not most other manufacturing methods. In this case you need to check that the shells do not intersect (they shouldn't or it means your model is broken anyway, nothing to fix) and if a shell lies inside another. You can do this by picking a point on the suspected interior shell and shooting a ray in a arbitrary direction. The shells that you hit a odd number of times are surrounding your main shell.


Path Tracing With Multiple Lights I've got my path tracer working great for a single light, but I'm having trouble coming up with a solution for scenes with multiple lights. My first thought was to just send a ray to each light and sum up all the contributions. That worked fine, but it was very slow, especially for many lights. I tried researching techniques for working around this, but I found very little. I did find that some renderers weight each light and then pick one based on how much light they might contribute, but I couldn't find any detailed process of how that's actually done.

So my question is, could someone give me an overview of how this might be accomplished, or point me to a resource that goes over multiple lights?

It's not that hard. If you have just planar or angular light sources, you can think of them as one light source split into multiple chunks and the only thing to deal with is how to sample this multi-light and how to compute the PDF of the resulting samples.
Picking probability
First, you need to setup the picking probability $P(l)$ for each light source $l$. The picking probabilities can be any non-negative numbers, you just have to make sure that are always non-zero if the contributions of the respective lights are non-zero. However, the closer the probabilities are to the actual (relative) light contribution, the better the overall performance of your Monte Carlo estimator will be.
Contribution estimates may be:

$1$: naive, uniform probability; good to start with; used by PBRT
$power$: better
$power / distance^2$: even better; closer light sources have greater contributions
…and other improvements: make zero if not visible at all, multiply by the cosine of the inclination angle, etc.

Note that to get the light-picking probabilities, you need to normalise the contribution estimates to the sum of all of them:
$$
P(l)=
  \frac
    {ContribEst_l}
    {\sum_{a\in\mathrm{Lights}}
      {ContribEst_a}}
$$
You could even use more different picking strategies if you find they work just for certain type of scenes and combine them using multiple importance sampling technique…
Sampling
When sampling the lights, you just select one light $l$ with the appropriate probability $P(l)$ and then sample the light surface/angle with its own sampling strategy (with PDF $p_l$). The resulting PDF $p$ will be:
$$
p(x)=P(l) * p_l(x)
$$
Evaluating PDF
When computing the light sampling PDF for given direction (as needed for multiple importance sampling), you just find the light source at the direction, evaluate the PDF of the lights own strategy and, again, multiply it by the probability of picking this particular light.
Conclusion
Now you have a sampling strategy which gives you PDF which can be used in the Monte Carlo estimators (e.g. $f(x)/p(x)$) and the resulting estimator will be unbiased.


Why shouldn't Bump, Normal and Displacement maps be gamma corrected? So I'm trying to wrap my head around this from a fairly technical point of view.

When you add a Bump or Normal or Displacement map in your shader they should not be gamma corrected. But diffuse textures should be. But why?

When you save a file in 8bit (or 16bit Integer) with a format like JPEG, PNG or TIFF they get the gamma of 0.4545 (1/2.2) burnt in. So a camera captures the light from the real world (that is linear) and when saving the photo as a JPEG a gamma value of 1/2.2 is added, as a way to compress the information into 8bit, and then when you view the image on a monitor that adds a gamma of 2.2 making the luminance linear again.

So when using a diffuse textures (like a photo) you need to a "remove" that 1/2.2 gamma to make it linear (called linearizing) so now the texture looks darker but these values are correct from a mathematical standpoint so the renderer will make the correct calculations. Side note: the software used a display gamma of 2.2 to make sure the textures looks correct it's just for the internal calculation that they are linearized. But when I create a Bump map in Photoshop and save that as a JPG or PNG then that 1/2.2 gamma is burnt into that image as well, so why doesn't this Bump map need to be linearized as well?

If I load a Bump map in 3ds Max with a input gamma of 2.2 just like a would do with a Diffuse texture, the diffuse looks right but the bump looks wrong, so I have to set the input gamma to 1.0 for the bump in order to make it look right, but as I said: why, both image files have the 1/2.2 gamma burnt in, right?

The fact that a photo is stored in gamma space doesn't have to do with PNG or JPEG, it has to do with the fact that it's a photo.

The camera detects a 50% intensity but is going to save it as a 187 instead of 127, because that is more efficient to keep the color information that matters to the human eye when you have only 256 different possible values. And so does Photoshop if you tell it so.

For normals or displacement maps, you don't have this need so there is no reason to use gamma space for them (using gamma for them would actually be a mistake, since it would affect how precision is distributed). Thus when you decode a normal texture or a displacement texture, you don't need to apply gamma.

If the normal map is encoded in a different space though, as can be the case for a G-Buffer for example, then you have to apply the corresponding correction.


How to do heterogeneous volume absorption using path tracing? I want to do volume rendering using path tracing with a absorption only model. And my data is in a 3D grid, which is turned into a voxel medium in my renderer.

However, I can't seem to be correct implementing the algorithm.

According to this note (however absorption-only heterogeneous case is not mentioned here), the radiance at point $x$, direction $\omega$, with emission-absorption model in a homogeneous medium is:

$L(x,\omega) = \int_{x}^{y} \tau(x,x') \epsilon(x')dx + \tau(x,y)L_e(x,\omega)$

where $\tau$ is the attenuation, y is a point on surface, and $\epsilon$ is emission at point $x'$.

So I come up with the heterogenous, absorption-only equation as:

$L(x, \omega) = \tau(x,y)L_e(x,\omega)$

since I think the emission term should be omitted. However this equation seems not correct intuitively and do not give correct image.

According to this equation, the radiance at every point $x'$ along the ray within the volume is not counted, which is where I think the problem lies.

So what is the correct heterogeneous volume path tracing equation?
How to derive the correct equation?

I give more details of my implementation here. The scene is simply a cornell box where only a voxel medium is inside the box. I trace a ray from the eye(camera), and the ray go through the medium and intersect a surface(the wall). To simplify the procedure, I set max_bounce to 1 which means the path is
eye -> medium -> surface -> light.

So the value of this sample is simply the throughput of the medium times the throughput of the surface times direct light at surface intersection.

However I got the result image

Your equation for the absorption-only model is correct: You multiply the luminance $L_x$ at the surface by the transmittance of the participating media between eye and the surface. For homogeneous media you can calculate this analytically using Beer-Lambert law:
$$L_y=L_xe^{-\mu|x-y|}$$
For heterogeneous medium you can calculate the transmittance by ray march through the medium using for example Woodcock tracking explained in the paper you referred.


How mesh geometry data (vertex coordinates) stored in glTF? I've read the reference guide about the file but I am really confused. Why the hell are there "min" and "max" values ?? A single vertex has only one coordinate array not min and max..

Also in the structure of the meshes why is there an "indices" reference to the "accessors" if we already put the indices to the "attributes" like "NORMAL: 2"?

The min and max values provide an outer bounding box for all POSITION data within a given accessor.  Accessors typically contain multiple vertices, for example all of the vertices of a particular primitive, and then min: [x, y, z] and max: [x, y, z] will offer the bounding box for that primitive.

For your second question, let's look at the mesh structure from BoxTextured.gltf:

"meshes": [
    {
        "primitives": [
            {
                "attributes": {
                    "NORMAL": 1,
                    "POSITION": 2,
                    "TEXCOORD_0": 3
                },
                "indices": 0,
                "mode": 4,
                "material": 0
            }
        ],
        "name": "Mesh"
    }
],

In the above example, mode: 4 is an enum, where 4 means TRIANGLES.  The indices are stored in accessor 0, and the POSITION data is stored in accessor 2.  So, accessor 2 will contain a list of vertex positions, but each vertex is allowed to be referenced by multiple triangles.  The indices in accessor 0 declare the triangles by indexing into the list of vertices.

Here's a sample of the start of accessor 0 from this model:

0
1
2

3
2
1

4
5
6

7
6
5
...

And this is a sample of the contents from the start of accessor 2 in this model:

-0.50000    -0.50000     0.50000
 0.50000    -0.50000     0.50000
-0.50000     0.50000     0.50000
 0.50000     0.50000     0.50000
 0.50000     0.50000     0.50000
 0.50000    -0.50000     0.50000
 0.50000     0.50000    -0.50000
 0.50000    -0.50000    -0.50000
 ...

In this manner, the sample model builds a cube by winding triangles around vertices at the corners.

For a more graphical explanation of this, check out the glTF Overview Card.


How can an OpenGL visualization run for weeks? For an art project, I need to make an OpenGL visualization run for at least 4 weeks. Is this possible? I can pretty much choose the rest of the hardware and software.

Opengl doesn't care how long it has been running.

However there are a few other challenges with long running applications in general:

memory leaks, both normal ones and opengl resources.
lack of precision after incrementing a float to a big number. After 4 weeks your millisecond counter will be at 2.4 billion which is well beyond the point where integers cannot be expressed exactly in a single precision floating point number. Make sure that those numbers are wrapped down to a smaller value explicitly.


Matlab scatter hotmap I have two columns of data, x(angle) and y(distance), they are my experiment results(figure 1). How can I use these results to make a ‘hotmap diagram’ by matlab? The z axis is the density value of xy dots in the 2D coordinate department, like below figure2. Thank you!

You basically need to plot the scatter map with varying colors for each point

scatter(x, y, [], c);

The main difficulty is to calculatec so that it’s proportional to the density.

To do so you can divide the map into a grid of a certain resolution, say [res_x, res_y]. Then count for each grid cell how many points are there. This count is an approximation of the density at that grid cell. Finally you can assign to all points within this grid cell the c value equal this count.

Probably you also need to normalize c to [0, 1]


2D sampling with multidimensional transformations I'm currently learning mathemetical concepts of distribution and the way to use them in a ray tracer with the book "Physically Based Rendering".

Let's start by uniformly sampling an hemisphere:

As you probably know, a way to generate the uniformly distributed direction is to use the inversion method.

Let us denote by $p$ our uniform probability density function:

$p(\omega) = \cfrac{1}{2\pi}$ and so $p(\theta, \phi) = \sin(\theta)p(\omega)$.

Then you compute $p(\theta)$, $p(\phi | \theta)$, you integrate your cumulative distribution function and you invert the function.

My questions are:

What does $p(\theta, \phi)$ really mean?
What is the transformation between $p(\omega)$ and $p(\theta, \phi)$?
In the book, to find $p(\theta,\phi)$ they state that $p(\theta, \phi) d\theta d\phi = p(\omega)d\omega$, but why?

I know that for $p(\omega)$, our random variable is a given $\omega$ (a direction), so the function represents a relative probability for this direction (so a solid angle, because the relative term implies a direction and a delta area around this direction).

But for $p(\theta,\phi)$, our random variable is now the couple $(\theta,\phi)$. To what extent is it different from a direction?

I'm not sure I've correctly understood the question, but here goes.

You're trying to sample directions uniformly, so you've got $p(\omega)$, which is the probability of getting a particular direction. But what is a direction? You actually need your probability distribution to produce numbers in some representation, and the easiest representation to deal with is lat-long (i.e. two angles). So the thing you actually need to sample from is the probability distribution of pairs of angles. This is what $p(\theta, \phi)$ is: the joint probability of two variables.

$p(\omega)$ and $p(\theta, \phi)$ mean the same thing geometrically, but the former gives you an abstract direction you can't sample from directly, while the latter more usefully gives you two numbers that represent a direction.

The reason for your third bullet point is to do with the point you've made about how it isn't just a single direction. These aren't really functions: they're distributions. A direction is infinitesimal, so you can't have a probability of just one direction. What you actually need to do is integrate it over the directions you're interested in.
$$
\int p(\omega)d\omega = \int_0^{2\pi}\int_0^\pi p(\theta, \phi)d\theta d\phi = 1
$$

Whichever representation you use, the integral over the hemisphere has to be 1, because it's a probability distribution.

I'm not sure if I need to explain this or if you already understood, but here's the origin of the $sin(\theta)$. When you do the double-integral on the right, you're splitting the problem up into a series of rings, or slices of the unit sphere. Each ring has constant $\theta$ while $\phi$ goes from $0$ to $2\pi$. Also, the area of each individual ring decreases as $\theta$ increases: the ring at the equator is huge, while the last "ring" at the pole is tiny. The area decreases as $sin(\theta)$. Because we want each unit of area of the sphere to have the same probability, we need the smaller rings to get a smaller share of the probability.

As Florian R. explains, you can do that by including the $sin(\theta)$ factor in the integral, or you can put it inside the definition of $p(\theta, \phi)$ like the book does.


Two questions on Gamma correction and Tonemapping I have few questions regarding tone-mapping and gamma correction.

is a tone-mapping operator a function whose output is a value between [0,1] ?
If this isn't the case and since gamma encoding requires a value to be in range [0,1], do I need to normalize the values after tone-mapping?
imagine a diffuse horizontal surface with (0.5,0.5,0.5) reflectance under an unobstructed uniform sky of radiance (1,1,1) cd/m2. The incident radiance on an image pixel coming off this surface will have value of (0.5,0.5,0.5). Now, most codes I have seen apply gamma encoding(raising pxl value to the power of 0.45) at this stage just before saving it to the file. But on viewing the image, the colors are distorted and much brighter corresponding to (186,186,186) color value. If I don't apply gamma encoding then the image file displays the expected color that is (127,127,127).

If the CRT monitor or the LCD does apply the inverse of gamma, then by encoding it I must see the original color that is (127,127,127), is that right?
And if I am seeing (186,186,186) instead, am I right therefore to assume the encoding happens internally on saving the image data as png or jpg format? and therefore I don't need to do any gamma encoding?

the colors are distorted and much brighter corresponding to (186,186,186) color value. If I don't apply gamma encoding then the image file displays the expected color that is (127,127,127).

No, the gamma-correct color is not distorted; your expectations are. If your lighting, meshes, textures, and other scene elements were not built to be properly linear and gamma correct to the screen, then every decision you have made has been through a distorted lens. If you don't like the effect you're getting with a properly gamma-correct lighting system, that's the fault of all of those decisions you've been making, not with gamma correction.

Half of the full lighting intensity in a linear colorspace does correspond to greater than half in a gamma-correct colorspace. And those gamma-correct values are what you ought to be passing to the screen. If that's not what you're shoving into the display, then you're doing lighting wrong.