Efficiently transferring only needed parts of the vertex data to the GPU (C++/OpenGL) I have a situation, where I have an array of vertices on the CPU side, for which, besides rendering-relevant data, like their position, I have a lot of other data at each vertex. All vertex data is updated (more or less) every frame, requiring transferring the changes over to the GPU every time.
So far I have used two approaches, neither of which I am particularly happy with:
Approach 1
Just copy everything over to the GPU. This might seem silly, but my application does not use a significant amount of VRAM overall, so this is no issue in that regard, but of course, when the vertex count increases, so does the amount of data needlessly transferred to the GPU every frame, which does become a performance issue.
Approach 2
Keep separate arrays for the GPU-relevant vertex data and the non-relevant vertex data. While this solves the unnecessary transfer issue, I find it very clunky to work with on the CPU side, since I at all times have to keep two arrays in sync. Ideally, I would like to have one clean struct per vertex, which I can easily work with.

Is there some other approach to this?
Something I could think of, which unfortunately doesn't seem to exist in OpenGL, is a variant of glBufferData, which I could tell some stride and offset, like with glVertexAttribPointer, which parts of each data block I want to transfer to the buffer in VRAM.
There also might be some way to neatly solve this with some C++ features that I am not aware of. Some way to have two arrays act as one, or something like that.
Any ideas and suggestions are appreciated.

This more or less a long comment than an answer.
Some other options that come to mind are the following:

A little old school, but you can create an interface with one of the texture objects. You can query different mipmap levels for your needs inside the shader. Textures are one of the better cached objects in OpenGL.

You can defer the lightening process by sending everything to a framebuffer with multiple rendering targets. The first pass would filter out the relevant parts of your data and create a geometry buffer. You can use the g-buffer for lightening pass afterwards.

You can create a flat array where your relevant and non relevant data are divided in equal strides, then use the indices of the relevant parts stored in your Element Array Buffer object in glDrawElements call. Basically you can use something like the following:

struct MyRealData {
vec3 myRelevantData
float myNotSoRelevantData
};
void to_drawable(MyRealData[20] ds, vec3[40] flat_drawable_array){
  for (int i = 0; i < 40; i+=2){
    flat_drawable_array[i] = ds[i].myRelevantData;
    flat_drawable_array[i+1] = vec3(ds[i].myNotSoRelevantData, 0, 0);
  }
}
std::vector<unsigned int> indices[] = {
  0, 2, 4, // first triangle
  2, 6, 4, // second triangle
  4, 6, 8, // third triangle
// ... etc
};
.
. // later in the code
.
glDrawElements(GL_TRIANGLES, indices.size() , GL_UNSIGNED_INT, (GLvoid *)indices.data());

```


How to correctly render visible spectrum with uniform intensity? I'm trying to get the correct representation of visible spectrum to render on the screen. My current progress is with using CIE 1931 standard observer color matching functions analytically approximated by the equations given in this paper. Current image is like this:

Here the background is not ideally black to make it possible to introduce the negative contributions from the color matching functions:

But the problem with it is that the image seems to have a dip in intensity somewhere around 490 nm. I guess that this is related to the sum of the three functions having a minimum there. See the sum plotted against wavelength:

So my question is now: how do I fix the intensity of the image? Is there any standard for taking spectral sensitivity into account when converting wavelength to RGB/XYZ representation?

I guess that this is related to the sum of the three functions having a minimum there. See the sum plotted against wavelength

This is actually a wrong way to sum these functions: sRGB model specifies that to get luminosity one must weigh the RGB components with appropriate weights. Then luminosity, unsurprisingly, will become equal to the Y component of the XYZ representation.

  the problem with it is that the image seems to have a dip in intensity somewhere around 490 nm

I can actually see the dip not only near 490 nm, but also a smaller one near 570 nm.

Anyway, I think I've found the reason for this. First of all, let's plot the desaturated colors used in the rendering on the chromaticity diagram:

The dashed triangle is the sRGB gamut, and the cusp formed by the lower two lines of the green curve corresponds to the D65 white (gray) point. Notice that there are three "bumps" away from the white point, corresponding to violet, bluish-green and red. Or, that there is a large dip in saturation in the direction of 490 nm and a moderate decrease in the direction of around 570 nm. Let's characterize this graphically, plotting the norm of difference of the rendered points from white point:

This is exactly what I perceive visually in the image in terms of saturation and thus blending with the background.

Where does this strange shape come from, you may wonder. And the answer is, from the spectral locus in full XYZ space. See the following figure: the blue curve is the spectral locus, and the orange is its projection onto a plane normal to vector {1,1,1}.

So, the shape we got on the chromaticity diagram is something like a planar unfolding of the spectral locus.

As to realism of this rendering, I'm reasonably sure that if we try projecting real spectrum onto a gray non-fluorescent surface, we'll also see these two "strange" dips. And they will become less noticeable as you increase brightness of the spectrum relative to the brightness of the surface.


What does $E_L$ mean in the context of shading equations from the book "Real time rendering"? We will use $E_L$ for irradiance perpendicular to $l$

Where $l$ is the light vector.

What does this mean? How do I calculate $E_L$?

The line you quoted says it: $E_L$ is the incoming irradiance at the surface due to the light source under consideration. Less technically, it's a vector representing the intensity and color of the incoming light at that point.

The equation you mention is for a punctual light source (directional, point, or spot light), so in this case $E_L$ would simply be the user-defined RGB color of the light source, multiplied by any relevant attenuation factors—such as shadows, distance attenuation for point/spot lights, and cone angle attenuation for spot lights.


Bezier Curve Tool Controlled by Mouse So I have recently learned about Bezier Curves (quadratic & cubic) and have a thorough knowledge of how they work. In my graphics engine in DirectX I wanted to implement a Bezier Curve Tool exactly like the Pen Tool in Adobe Illustrator. I understand the basic computation of the curve but I am puzzled as to how I can get mouse interaction involved.

Basically I'd like to draw Bezier Curves with a Mouse.

Instead of using cubic Bezier spline, you should use cubic Hermite spline. For cubic Hermite curve you specify the segment end points and tangents, which is what you then control in the tool. Hermite spline is almost the same as Bezier spline but with different $B$ matrix, so if you already understand Bezier, then understanding Hermite is a trivial step.

So in the Adobe Illustrator pen tool (which based on the video I take is the same as Adobe Photoshop pen tool that I have used) you keep adding more Hermite cubic spline segments to your spline. By default the Hermite start tangent matches (with opposite sign) the end tangent of the previous segment and in GUI they are linked together (changing one changes the other). So in that configuration the curve appears smooth. However, you can unlink the tangents to cause C1 discontinuity in the spline and control them separately. This is purely GUI work though and how you control the Hermite spline tangents & positions.


How to get camera Right vector when Front vector and WorldUp vector are collinear Get the title problem when implementing a walkthrough camera.

I use normlize(cross(WorldUp, Front)) to get Right vector. And when they are collinear, I get a zero vector.

Is there any more robust fake version? Or do I need to store the "previous" Right vector as a backup?

You can't have a camera with collinear look/front and up vectors. It doesn't make geometric sense. The whole point of the up vector is to disambiguate roll.

As russ says, for a "first-person" camera it's reasonable to always use world-up for the up vector and simply lock the pitch angle to within (say) 85 degrees of horizontal, to avoid degenerate cases.

For a CAD viewer where the user would expect to pitch the camera arbitrarily, you should rotate both the up and front vectors about the right vector when the user pitches the camera, to ensure they are always perpendicular. Beware that when you do this kind of arbitrary rotation, it's easy for the user to get stuck in unfortunate roll angles. You need to do at least one of the following to let them get themselves out:

Have a "reset up" button that applies the pitch angle limits and resets the up vector to world up.
Give the user arbitrary control over camera roll. (This can be confusing, so it's best to offer the first option as well if you do this.)
Constrain the up vector so that its dot product with the world up is non-negative. That is, if the user pitches so far up or down that they are upside-down, roll the camera 180 degrees.


How to solve Z buffer artifacts I am currently rendering a scene in the most standard fashion possible (triangle projection and Blinn-Phong for shading)
However I am getting some artifacts:

As you can see the lion head is not rendered properly. Moreover, the artifacts are flickering, but the proportion of how much they flicker is proportional to the distance to the lion. That is to say. If you are close to the lion there are no artifacts. But the farther away you go the more artifacts appear. I believe this is due to Z buffer imprecision. How can I solve this?

Short answer: Move your near clip plane further away.

Depth buffer precision is very sensitive to the near clip plane distance.

Complicated answer: Use different math in your view projection. There are a few techniques that can help, some of them are outlined here:

https://developer.nvidia.com/content/depth-precision-visualized


What is the correct order of transformations scale, rotate and translate and why? This is a rather primitive question coming from an electronic engineer. When applying rotate (about origin), scale (in which we shall translate towards origin and then back) and translate, does it matter in what order we do it? Why?

Basically in my case I have an image in coordinate space that goes from -2 to +2 in x and -1.5 to +1.5 in y to get ration of 4:3. This range comes from how the original mandelbrot set is plotted. I need to scale and translate this to fit into an axes (of pixels) in bitmap that goes from 0 to 800 in x axis and 0 to 600 in y axis. I am trying to understand what matrix to use to scale and translate the points from my mandelbrot set into the bitmap image.

Usually you scale first, then rotate and finally translate. The reason is because usually you want the scaling to happen along the axis of the object and rotation about the center of the object.

In your case you don't really need to worry about this generic solution though, but you only need to map range [0, 800] $\rightarrow$ [-2, 2] for x-coordinate and [0, 600] $\rightarrow$ [-1.5, 1.5] for y-coordinate, in order to map screen coordinates to real/imaginary components for Mandelbrot calculation. So this is simply done by:
$$real=4*(x+0.5)/800-2$$
$$imag=3*(y+0.5)/600-1.5$$

Note that you need to calculate Mandelbrot coordinates from screen coordinates and not the other way around. This is to ensure that you evaluate the Manderbrot equation for each pixel exactly once and are not left with holes in the image or do double evaluation per pixel.


Find closest point on surface interpolated by 2d matrix of points I have rectangular matrix of 3d points sampled from a parametric surface. In my program this matrix provides base points for bilinear interpolation of surface (In any (u,v) point it returns 3d point interpolated from neighboring sampled values). Now I want to find closest point on surface from any other point in 3d space, considering values "between" sampled points of grid.

I'll attach picture with example on 2d curve. Here yellow points are sampled points, red point - point to which we want to find closest, and green - point which we need to find as closest (witness point).

This is only a rough idea, though I hope it may inspire better answers.

The closest point on a surface is as far as I know always forms together with the red point a line that is orthogonal to the surface.

If you know which yellow vertices form the surface that the green point lies on, this is easy. You could just formulate the normal form of the surface (which is piecewise linear so it works out) formulate the orthogonal line to the surface that goes through the red point and solve for the intersection point. Though I don't have the exact math here, this step should be easy to look up.

Now the problem is: Which linear piece of your surface is the closest one.
Either you calculate the green-point candidates for all linear pieces and also the distance and take the closest one, or you find some other possibly faster way for this.
For that I don't have a more clever idea.


Return x; y; coordinates of vertex C of triangle I would like to return the XY coordinates of vertex C using math formula.
I have 5 input data what I always know and can be used for solution (green in the image): I know X; Y; coordinates of point A and point B (Ax, Ay, Bx, By), and I know the length of all three sides of triangle (AB, AC, AD). Please help how to get the values of X Y coordinate of point C, what is 70/30 in the example, but I have no formula to get back them.

I know that more than one solution of this formula, but:

Coordinates cannot be in negative areas. All coordinates must be positive.
I always use 1 direction for vertex labelling, but because I always know the length of AC and BC, I think to change the label from A to B will not change the position of C.

Thanks!

You can get the coordinates of C by using circle equations for one circle C_A with point A as center and AC as its radius and one circle C_B with point B as its center and BC as its radius.

Now you can calculate their intersection points, which leaves you with 2 points (iff C is not on a line between A and B).

I am not entirely sure how you want to determine which of the two candidates C_1 and C_2 is the correct one. If I understand correctly, you always label the triangle counterclockwise. There is a neat way to check this, so you should have everything to find your point.


Multiple buffers and calling glBufferSubData In my project, for convenience I would like to use many buffers.
Many buffers in my case means 50-100 terrain patches represented by buffers with vertex coordinates, normals, indices and maybe color. The magnitude of data would be, let's say 10^4 floats per buffer. Some of this data can be shared between each terrain patch, f.ex. xz-coordinates and indices.

During the rendering loop, some terrain patches will be updated. Which means that for certain buffers I call glBufferSubData() for the whole buffer.

My question is; are there any pros/cons, performance wise between these two methods:

1) Controlling my data in many buffers (50-100), thus letting me call glBufferSubData on a complete buffer when needed.

2) Controlling my data in fewer (5-20) buffers, with more data in each. But then having to set up a system where I need to call glBufferSubData on smaller portion of a buffer. (Which leads to a more complex design in my case).

It depends. There are a few competing factors at work here.

First, consolidating terrain data into fewer buffers (or one) may allow you to combine multiple terrain patches together in a single draw call—assuming there aren't other state changes between patches that would prevent this. So, you may be able to reduce draw-call overhead this way, or at least reduce the state-change overhead between draws. That being said, 50–100 draw calls is not that many, so this may not be a real performance concern.

In addition, do you anticipate needing to use the terrain data for other shader effects? For example, maybe a shader procedurally generates grass growing on the terrain, or you use height above the terrain to influence lighting or AO or some such. In that case having all the terrain data in one big buffer that you can pass to those shaders may simplify things.

On the other hand, if you're having the terrain deformed/edited in real-time (which it sounds like you are), then it may be faster to update the entirety of a small buffer than to update a small portion of a large buffer. The problem is when the CPU and GPU are working asynchronously, and you update a buffer from the CPU, the GPU may still be using the previous version of that buffer. To fix this, the driver internally keeps 2–3 instances of the buffer (much like double/triple buffering the screen). If you always update the entire buffer at once, the driver can just put the updated data into the "next" instance of that buffer. But if you update only part of the buffer, the driver must copy all the rest of the data from the current instance to the next instance. This extra copy will consume some performance either on the CPU or on the GPU.

So, if the terrain data was entirely static I'd say it's better to put it all in one big buffer. However, if you're going to be updating portions of the data in real-time, it may be better to keep it in smaller buffers so that you can update exactly the ones that have changed while leaving others alone.

In the end, you'll have to do some profiling to see which approach is better.


How to access several textures from huge amount of textures in a shader? I'm developing a light probe based GI solution and currently dealing with light leaks due to interpolation between probes located on opposite sides of the walls. I want to tackle the problem by incorporating precomputed shadow maps assigned for each probe to determine which ones are occluded and therefore which probes need to be interpolated for the current fragment in the fragment shader.

The question is, how would I even store tens of thousands of shadow maps and access them in a fragment shader?

Maybe store them continuously in a huge buffer texture? Hmm, or maybe pack them to a huge texture atlas? That also makes me wonder which method would provide a better cache behavior..

Well, it's unlikely that 10,000 cubemaps could fit into memory at all (at 128x128x4-bytes-per-pixel, 10,000 cubemaps would require upwards of 4GB of RAM). But as far as the mechanism to use a large number of such textures, that's pretty easy. You have two options.

The option that's widely available is to use cubemap array textures. You allocate the storage for the array cubemap with the number of cubemaps you want * 6. Each set of 6 array layer-faces is a single cubemap. In your shader, you just provide an array index for the cubemap layer (not layer-face) that you want.

The other option is to employ bindless textures. You create however many cubemaps you want, then provide them to the shader as an SSBO array of 64-bit integer handles. You index this array as needed and convert the handle into a sampler value before fetching.


Is gamma still important to take into account? Do different monitors (including mobile screens) still use significantly different gamma functions when displaying colour images? Is there a standardised way to represent colour that can then be translated according to the gamma of the monitor, or are they all sufficiently similar to preclude the need for this nowadays?

Yes, while many screens and OS operations are using a gamma of 2.2 your hardware and computation result still need to be corrected. There are also special mediums such as broadcast TV's that have a different gamma. Sensor equipment like cameras are mostly linear so they need to be adjusted accordingly. Besides this the user can set their system to whatever gamma they like.

In computer graphics primary reason to account for gamma is that your computation is most likely done in linear space* or summing light contributions becomes unnecessarily complicated. In any case gamma is really a simplification of things, you'd be much much better of doing profile to profile conversions if its possible to invest the computational time to do so.

Mathematical explanation

This may be a simpler to understand. Your monitor has a display error of $g(x)$ which is no problem if your data is conforming to $g(x)$, however if you want to calculate:

$$ a+b+c+d $$

and $a$, $b$, $c$, $d$, have the error of $g(x)$ you would actually need to calculate.

$$ g(g^{-1}(a)+g^{-1}(b)+g^{-1}(c)+g^{-1}(d)) $$

You would need to do this for each sub element over and over again. Therefore instead of running the transform each time you transform all once to linear and then once back.

And finally a artistic reason

It may just look better with a different gamma. Many games have a gamma adjustment for this reason.

* Worst case is that you think your computations as linear but do not compensate for the monitors nonlinear output charachteristics.


Best way to clear a framebuffer image without drawing? So I have a framebuffer image which later acts as input to a compute shader, and some frames there is geometry drawn to the framebuffer, but some frames this is not necessary.

The thing is, in my current process, when I don't draw anything to that framebuffer, the image contents are undefined which leads to incorrect output of my compute shader.  So on frames where there is no geometry drawn, I would like to simply clear the framebuffer.

In OpenGL this would be easy: glClearColor(...) but so far I have not been able to get it working on Vulkan.

I have tried just doing an empty renderpass: i.e: vkCmdBeginRenderPass/vkCmdEndRenderpass with the correct clear values defined in the VkRenderPassBeginInfo struct.

I have also tried doing a renderpass with only vkCmdClearAttachments between the bein/end commands.

So far I haven't had any luck.  What's the best way to accomplish this?

You are looking for vkCmdClearColorImage,

It does exactly what it says on the tin, it clears an image to a specified color. This command uses the Transfer stage as far as barriers are concerned. You will need to add one between the clear and the vkCmdDispatch.

There is also a vkCmdClearDepthStencilImage if you need it.


Is there a image-format that support an array per pixel? For a project dealing with deep semantic labeling of images I need a format that supports arrays of arbitrary length for each pixel of the image.

The array is an array of integers.

Which image formats support this?

OpenEXR might be a good fit. You can have an arbitrary number of channels per pixel and the data types for each channel can be integers.

http://www.openexr.com/


How to send shadow rays to window light source with glazing For rendering an indoor scene where light source is skylight coming in through
the window opening, one can consider the window polyline as the light
source, with the sky distribution, in direct lighting calculation. And sample
the window area(or solid angle).

It works as long as it's just an opening. Once a glazing is introduced I cannot figure out how to do direct lighting(in a path tracer) correctly. Shadow rays either are transmitted or reflected. I guess the transmitted ray needs to be traced again but this is not easy I suppose(as the the Brdf.SampleDirection method returns only direction), considering also that it might be a double
glazing which inter-reflection can happen too.

Does any one have any suggestion or experience on this?
Thanks,

Most renderers allow you to set a flag on an object to make shadow rays ignore it (so it won't cast shadows). You could set this flag on your glazing.

That said, it's not unusual to make it so that refraction is disabled on shadow rays: that is, if they would be transmitted by an object, a child ray is traced in the same direction, ignoring any refraction that would be caused by the ior of the material. Although this gives wrong results for objects such as crystal balls, you need photon mapping or similar to get good results for that situation anyway. Just tracing a child ray is an easy way of making objects like stained-glass windows give simple coloured shadows.


Does an object spanning percentage of the Camera's FoV correlate with the object's image percentage? I am working on a virtual camera inside a game engine, which provides me with an FoV control, but no Focal length control (and no filming sensor dimensions short of aspect ratio), and I am less than a beginner in photography.
Now when I position the Camera in a specific location in the world, and I position an object in front of the camera such that the object covers 50 degrees of the camera field horizontally. The camera FoV is 100 degrees horizontally.[see image please].

Then I measure the object in the image and I find it covering around 40% of the image width instead of 50%.
What is the most expected mistake that I am making here?
Relevant information: I don't know the position of the virtual camera is the position in the middle of the lens itself, the position of the sensory retina, or the position in between.

@ Physician --- We can use trigonometry and set-up a condition whereby the horizontal angle of view will be 100°
On a full frame 35mm camera format demotions 24mm by 36mm, mount a 15mm lens and horizontal angle of view will be 100°
On an APS-C compact, the frame size is 16mm by 24mm. Mount a 10mm lens and the horizontal angle of view is 100°
In other words, the angle of view of camera system intertwines focal length and format dimension.
For the full frame camera above:
The image plane falls 15mm downstream from the lens.  The Image triangle has a height of 15mm. The size of the angle at the apex of this triangle is 50°. We bisect this triangle creating two right triangles. The angle at the apex is 25°. The height of this triangle is 15mm. If you draw this triangle, the height is the adjacent side and the base is the opposite size. The tan of 25° = 0.4663. Thus Opposite / Adjacent = 0.4663. Thus Opposite side = 15mm X 0.4663 = 6.9946mm. The length of the image of the object will be 2X this value = 14mm.
For the APS-C format:  The image plane falls 10mm downstream. Such as lash-up delivers a 100° horizontal angle of view. The image triangle angles are the same as for the full frame. Bisecting the image triangle, we get two right triangles each with a 25° apex. The tan of this angle = 0.4663. The height of this triangle is the focal length of the lens = 10mm. 10mm X 0.4663 = 4.66mm X 2 = 9.33mm. The image of the object will be 9.33mm long.


Wondering if it's possible to 'extend' a texture beyond the mesh without altering the mesh itself I'm in Blender and working with a pre-existing 3D model- I want to add more length to it, but I don't want to mess with the mesh- Is it possible to extend the texture beyond the mesh without changing it, or will I have to change the mesh anyways? Are there any shortcuts around changing the mesh?

Textures are applied to the faces of a mesh, so textures cannot be anywhere there isn't a face for it to appear on. "Adding more length" to a mesh pretty much implies moving some of the vertices. I don't think you're going to get around that.
Now if you are assembling a game world, it's quite common to have modular mesh pieces that can be aligned end-to-end, or placed intersecting each other, to form larger structures—like a wall panel that can be duplicated several times to form a longer wall, for instance, without needing a new mesh to be built specifically for that wall.


How to convert Euler angles to Quaternions and get the same Euler angles back from Quaternions? I am rotating n 3D shape using Euler angles in the order of XYZ meaning that the object is first rotated along the X axis, then Y and then Z. I want to convert the Euler angle to Quaternion and then get the same Euler angles back from the Quaternion using some [preferably] Python code or just some pseudocode or algorithm. Below, I have some code that converts Euler angle to Quaternion and then converts the Quaternion to get Euler angles. However, this does not give me the same Euler angles.

I think the problem is I don't know how to associate yaw, pitch and roll to X, Y an Z axes. Also, I don't know how to change order of conversions in the code to correctly convert the Euler angles to Quaternion and then convert the Quaternion to Euler angle so that I am able to get the same Euler angle back. Can someone help me with this?

And here's the code I used:

This function converts Euler angles to Quaternions:

def euler_to_quaternion(yaw, pitch, roll):

        qx = np.sin(roll/2) * np.cos(pitch/2) * np.cos(yaw/2) - np.cos(roll/2) * np.sin(pitch/2) * np.sin(yaw/2)
        qy = np.cos(roll/2) * np.sin(pitch/2) * np.cos(yaw/2) + np.sin(roll/2) * np.cos(pitch/2) * np.sin(yaw/2)
        qz = np.cos(roll/2) * np.cos(pitch/2) * np.sin(yaw/2) - np.sin(roll/2) * np.sin(pitch/2) * np.cos(yaw/2)
        qw = np.cos(roll/2) * np.cos(pitch/2) * np.cos(yaw/2) + np.sin(roll/2) * np.sin(pitch/2) * np.sin(yaw/2)

        return [qx, qy, qz, qw]

And this converts Quaternions to Euler angles:

def quaternion_to_euler(x, y, z, w):

        import math
        t0 = +2.0 * (w * x + y * z)
        t1 = +1.0 - 2.0 * (x * x + y * y)
        X = math.degrees(math.atan2(t0, t1))

        t2 = +2.0 * (w * y - z * x)
        t2 = +1.0 if t2 > +1.0 else t2
        t2 = -1.0 if t2 < -1.0 else t2
        Y = math.degrees(math.asin(t2))

        t3 = +2.0 * (w * z + x * y)
        t4 = +1.0 - 2.0 * (y * y + z * z)
        Z = math.degrees(math.atan2(t3, t4))

        return X, Y, Z

And I use them as follow:

import numpy as np
euler_Original = np.random.random(3) * 360).tolist() # Generate random rotation angles for XYZ within the range [0, 360)
quat = euler_to_quaternion(euler_Original[0], euler_Original[1], euler_Original[2]) # Convert to Quaternion
newEulerRot = quaternion_to_euler(quat[0], quat[1], quat[2], quat[3]) #Convert the Quaternion to Euler angles

print (euler_Original)
print (newEulerRot)

The print statements print different numbers for euler_Original and newEulerRot which I don't want to be the case. For example if euler_original contains numbers like (0.2, 1.12, 2.31) in radians I get this Quaternion --> [0.749, 0.290, -0.449, 0.389] and converting the Quaternion to Euler angles gives me this --> (132.35, 64.17, 11.45) which is pretty wrong. I wonder how I can fix this?

Although I'm interested in getting the above code to work by making changes to it but, I would rather learn how to set up the equations correctly. This way I would know how I can get the correct Quaternions even if the order of rotations (XYZ --> YZX etc) for applying Euler angles is changed.

I had to change the order of inputs to def euler_to_quaternion as follow and have to convert the inputs to radians:

def euler_to_quaternion(roll, pitch, yaw):

        qx = np.sin(roll/2) * np.cos(pitch/2) * np.cos(yaw/2) - np.cos(roll/2) * np.sin(pitch/2) * np.sin(yaw/2)
        qy = np.cos(roll/2) * np.sin(pitch/2) * np.cos(yaw/2) + np.sin(roll/2) * np.cos(pitch/2) * np.sin(yaw/2)
        qz = np.cos(roll/2) * np.cos(pitch/2) * np.sin(yaw/2) - np.sin(roll/2) * np.sin(pitch/2) * np.cos(yaw/2)
        qw = np.cos(roll/2) * np.cos(pitch/2) * np.cos(yaw/2) + np.sin(roll/2) * np.sin(pitch/2) * np.sin(yaw/2)

        return [qx, qy, qz, qw]

Also, quaternion_to_euler was returning degrees and I had to change it to radians:

def quaternion_to_euler(x, y, z, w):

    t0 = +2.0 * (w * x + y * z)
    t1 = +1.0 - 2.0 * (x * x + y * y)
    roll = math.atan2(t0, t1)
    t2 = +2.0 * (w * y - z * x)
    t2 = +1.0 if t2 > +1.0 else t2
    t2 = -1.0 if t2 < -1.0 else t2
    pitch = math.asin(t2)
    t3 = +2.0 * (w * z + x * y)
    t4 = +1.0 - 2.0 * (y * y + z * z)
    yaw = math.atan2(t3, t4)
    return [yaw, pitch, roll]


how to build the feature curves of this algorithm? I hope this is the right group to be able to expose my doubts about the implementation of the algorithm shown in this article. The question I have is how to do the feature curves, what I do not understand is how to use piecewise Bézier cubic spline to build them. I hope someone can help me with an example please to be able to advance with that algorithm. Any help is welcome, thanks in advance.

The way I read it, you have a 2D cubic Bézier that defines the projections of the ridge line (or river bed, or whatever) onto the X/Y plane. In Figure 4, these are the dark purple lines.

So that gives you the direction that the ridge line moves. Now you have to define the height of the ridge line. This is done by picking some points along the 2D Bézier and setting the following values at each of the chosen points:

h = height
r = radius (this appears more like a width at the ridge line than a curved radius)

You can then also set the angle of the terrain as it comes off the radius. These are:

`phi` and `theta`
`a` and `b` are the lengths of the slopes

Keep in mind that you don't have to set every value for every control point you pick. I assume any you don't set are defaulted to 0, or some reasonable default.

So for a hill, you would set h to the height of the hill at a few point. You would then set phi and theta to be horizontal, so yes, pi / 4. The length of a and b will decide how quickly the hill falls off as you move away from the ridge line.

It looks like the above 6 values are turned into another cubic Bézier curve that is perpendicular to the first one for the rasterization step in section 4 of the paper. h doesn't factor into the shape of the perpendicular Bézier because it's also perpendicular to it. So r, theta, phi, a and b are used to make a cubic Bézier. I believe that you can think about like this: the point on the ridge line where you define one of these control points is the link between 2 cubic Bézier segments - one to the left of the point and the other to the right. The images they show only show a single point between 2 Bézier curves, but the way I'm reading it there are actually 2 curves that look something like this:

So I think normally, you'd have 2 Béziers. One on each side of the control point they show in the paper. (Pardon my bad drawing.)  a would be the distance between control points 2 and 3 of the left curve. Theta would be the angle formed between the vertical and a of the left curve. r would be the distance between points 3 and 4 of the left curve. I'm unclear on how one would pick control point 1 in this scenario, though.

Ai and Ri have to do with the noise parameters and don't appear to me to be related to the Bézier curves at all.


How can I reproduce the MatchMoving/MotionTracking technique from Just Dance? I'm a software engineer student with a big projet idea but low graphic knowledge.

My goal is to compare two people dancing. One person would be the teacher with perfect moves and the other would be a dance trainee.

From my beginner point of view, all I need to retrieve is both skeletons with a motion tracking camera (like Kinect?) then I can do maths with the different reference points (X,Y,Z).

Is there a motion tracking camera + API or Library you would recommand me ?

I do not want to open a 3D software like MeshLab. I would like to retrieve the reference points through line of codes and manipulate them as I wish.

The Kinect SDK has a skeletal tracking demo in the c++ api (BodyBasics). The skeleton you get is the same "topology" unless it fails to acquire the skeleton. So you can easily compare the left shoulder of two captures as you always know which point it is.

You'll have to experience with metrics. A simple point to point metric would probably be sufficient though. Maybe with some weights on different body parts though.


How exactly does Gouraud shading apply color across a polygon? Does it do it by pixel or is it some other method? I'm a super beginner to GFX. So far I've learned about 3 shading models when rendering meshes
The first is Flat Shading. I seem to understand this, from my understanding, you take the normal of the face/polygon, apply your lighting model based on that, and the light angle, and other parameters and based on what you get, shade your polygon with  a single color, done.
I've learned about Phong Shading as well. Apparently Phong Shading takes the normals of the vertices of the polygon, and interpolates them for each pixel, thus sort of 'creating' a normal per every pixel, and shade accordingly. It is shaded per pixel, and as such you get smooth results
Gouraud shading is what I am most confused by. I referred to many online sources including this one and what I dont understand is how exactly the polygon is shaded? Is it shaded per pixel?
From my understanding, Gouraud is the same as Phong shading except that unlike Phong interpolating vertex normals, and creates normals  for every pixel and shading each pixel, Gouraud interpolates just the vertex normals of the polygon.
Im not quite sure what it does after that. Does it take these interpolated vertex normals and shade the polygon in a single color? Does it shade pixel by pixel just like Phong?

The difference is in the number of lighting calculations done. Where the lighting calculation produces some color.
Flat does one per face, uses the color across the face.
Gourand does 3 (for a triangle, one for each vertex), interpolates the color across the face.
Phong does a light calculation for every pixel/fragment (where the normal is interpolated across the face).


Why are these 2 buffers joint while extracting glTF information? I am using tinyGLTF to parse gltf files and then render them in my program.
I am currently during the third example in this page.
In particular I am trying to extract the 2 animation buffers:

My code:
MeshMap LoadGltf(const std::string& path)
{
    Model model;
    TinyGLTF loader;
    std::string err;
    std::string warn;

    bool ret = loader.LoadASCIIFromFile(&model, &err, &warn, path);

    if(!warn.empty()) { cout << warn << endl; }

    if(!err.empty()) { cout << err << endl; }

    if(!ret) { cout << "Failed to parse glTF" << endl; }

    MeshMap mesh_map;
    for(auto& mesh: model.meshes)
    {
        if(mesh.name.empty())
            mesh.name = "unnamed mesh " + to_string(global_mesh_index++);
        // TODO(low): implement this functionality as needed
        assert(mesh.primitives.size() == 1);
        GetPrimitiveData(model, mesh, mesh_map);
    }

    if(model.animations.size())
    {
        cout << "model.animations.size() " << model.animations.size() << endl;
        cout << "model.animations[0].channels.size() " << model.animations[0].channels.size() << endl;
        cout << "model.animations[0].channels[0].sampler " << model.animations[0].channels[0].sampler << endl;
        cout << "model.animations[0].samplers[0].input " << model.animations[0].samplers[0].input << endl;
        cout << "model.animations[0].samplers[0].output " << model.animations[0].samplers[0].output << endl;

        const int input = model.animations[0].samplers[0].input;
        const int output = model.animations[0].samplers[0].output;

        vector<float> time = ExtractDataFromAccessor<float>(model, input);
        for(auto f : time) cout << f << " ";
        cout << endl;
        cout << time.size() << endl;
        cout << model.accessors[input].name << endl;
        vector<float> rot = ExtractDataFromAccessor<float>(model, output);
        for(auto f : rot) cout << f << " ";
        cout << endl;

        cout << model.animations[0].channels[0].target_node << endl;
        cout << model.animations[0].channels[0].target_path << endl;
    }

    return mesh_map;
}

template<typename T>
std::vector<T> ExtractDataFromAccessor(
    Model& model,
    const int accessor_index)
{
    const int buffer_view_index = model.accessors[accessor_index].bufferView;
    const int buffer_index = model.bufferViews[buffer_view_index].buffer;
    const int buffer_length = model.bufferViews[buffer_view_index].byteLength;
    const int buffer_offset = model.bufferViews[buffer_view_index].byteOffset;
    const int buffer_stride = model.bufferViews[buffer_view_index].byteStride;
    const vector<unsigned char> data = model.buffers[buffer_index].data;

    //TODO(low): extend functionality to handle this as needed.
    assert(buffer_stride == 0);

    std::vector<T> holder(buffer_length / sizeof(T));
    memcpy(
        holder.data(),
        data.data() + buffer_offset,
        buffer_length);

    return holder;
}

Output
model.animations.size() 1
model.animations[0].channels.size() 1
model.animations[0].channels[0].sampler 0
model.animations[0].samplers[0].input 2
model.animations[0].samplers[0].output 3
0 0.25 0.5 0.75 1 0 0 0 1 0 0 0.707 0.707 0 0 1 0 0 0 0.707 -0.707 0 0 0 1
25

0 0.25 0.5 0.75 1 0 0 0 1 0 0 0.707 0.707 0 0 1 0 0 0 0.707 -0.707 0 0 0 1
0
rotation

I am very confused in that in and out seem to be equal, and that they are the concatenation of the 2 animation buffers, rather than what I expected. Which was in to be the time buffer and out to be the rotation buffer.
Where is the data that lets you separate them?

The accessor (model.accessors[accessor_index]) has its own byteOffset and count values that you need to account for here.
The input and output accessors are allowed to share a bufferView, and your output is showing a dump of the same entire bufferView twice.  The first five values in that dump are the input accessor with the times (in seconds, since the start of the animation) and the remainder of the bufferView is pointed at by the output accessor.  If you account for the additional settings on the accessor itself, it should allow you to pick apart this bufferView correctly.


Getting contrast color to a given color in RGB or ARGB or HEX I am looking for a stable way to find a contrast/inverted/opposite color to a given one.

Application of this is to find best possible color for foreground to give background.

I read this in a magazine article probably 25 years ago. I don't recall the full derivation, but they said to take each component (red, green, and blue) and invert the high bit of the color. In a C-like programming language, you could do:

 component = (~component & 0x80) | (component & 0x7F);

This can be simplified to:

component ^= 0x80;

for  a single component or to:

component ^= 0x80808000;

if you want to do 4 components at once (and don't affect the alpha).

This assumes your components are 8-bit values. This avoids the problems you get with the middle values if you simply invert them. In that case 127 becomes 128, and vice-versa. But with the above solution, 127 becomes 255 and 128 becomes 0.


Affine Transformation We have a matrix

$$\begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix}$$

This matrix represents a transformation in homogeneous coordinates.
My question is whether the above matrix is affine or not and an example too for this.

It is not necessarily affine. An affine matrix in homogeneous coordinates has a form like:

$$\begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ 0 & 0 & 1 \end{bmatrix}$$

(assuming you use a column vector convention). Here, the upper-left 2×2 submatrix is the linear part, and $(a_{13}, a_{23})$ is the translation vector of the affine transform.

If the lower row of the matrix has some values other than $[0, 0, 1]$, then it is in general a projective transform, not an affine one.


Why do polygons have to be "simple" and "convex"? I'm new to computer graphics and just wanted a solid understanding of why?

Why is it so important that polygons you push through the pipeline be "simple" and "convex"?

Polygon rasterization (the conversion of the analog polygon data into a raster image) is a key operation in rasterization-based rendering. As such, performing this operation fast, and with predictable performance, is key to the performance of graphics rendering.

It is possible to rasterize any arbitrary polygon. But rasterizing triangles is extremely simple to code and offers numerous efficiency improvements over general polygon rasterization.

Triangle rasterization is done by something called "scan conversion": you break the triangle up into a series of horizontal scan-lines (one pixel tall) of a length determined by the sides of the triangle. Building these scan-lines and getting their lengths is something you can do through basic integer math involving only additions and multiplications. Furthermore, each scan-line can be constructed independently of the others, so highly threaded hardware can be employed to perform this process efficiently.

None of this is true for arbitrary polygons. Concave polygons in particular require much more complex math. A concave polygon can have multiple lines in the same horizontal row.

Furthermore, interpolation of values across the surface of a triangle can be done by relatively simple math. For each scanline, you can compute a gradient that represents how much each value changes from one end of the line to the other. So given a starting value at one line, you can easily compute the interpolated position at any point on the scanline simply by multiplication and addition. If you're computing them in sequence, you don't even need the multiplication; successive addition is adequate.

Interpolation of values across an arbitrary polygonal surface is much more complicated.

Given all of these advantages, and the fact that you can use a series of triangles to build any arbitrary polygon, there's really no reason to rasterize any more complex shape.


Logarithmic spiral with equal vertex spacing, what equations? I need to draw a logarithmic spiral (or close approximation) whose vertices are equally spaced, such that the lines between any two consecutive vertices are of equal length.

(Actually, that spiral is just the basis for something a little more complex, but equally spaced incrementation is what matters right now.)

The basis for the spiral is
$$r=e^{0.25 \theta}$$
where $r$ is the radius and $\theta$ is the angle from the $x$ axis.

So it's easy to step around the spiral by a given angle, but obviously the actual length of any line between two vertices (defined by change in angle $\Delta\theta$) will vary with the radius.

How can I step along the arc of the spiral by a set length (say $0.1$ arbitrary units) and get the right coordinates (polar or Cartesian)?


Efficiently sampling specific surfaces using MIS in path tracing Apologies for posting two questions in one post, but as they are relevant I kept them in one.

Question 1

In the definition of multi sample estimator that Veach has defined, the integrand is the same for all the n sampling strategies. If this is so, then is it not the case that, in direct lighting, those shadow rays that miss the light geometry are to be considered, as the integrand is not zero (and the PDF is also not zero) but has some value, not from the light of course, but due to the surfaces that have occluded the light?

Would missing those contributions not be the same as partitioning the integral in sampling the lights and BRDF, whereas MIS claims to give
a solution without partitioning the integral?

Question 2

My main problem is to sample adequately part of the scene where light comes from the external obstructions (marked as P2 in the image). The only source of light is skylight coming in through the window opening. P1 is PDF for sampling window area (or solid angle) to get direct light; P2 is same PDF but to account for external surfaces contributions; and P3 is BRDF sampling for indirect light from indoor surfaces. I could rely just on BRDF sampling to account for external surfaces but as their contribution is large (illuminance from infinite source) I want to explicitly account for them.

I don't intend to partition the integral into three regions as it ends up to have recursive calls to get the contribution of external hits; it also slows down and makes the code too complex. I therefore guess I need to use MIS to account for indirect contribution of externals surfaces but am not clear how it is to be done.

I tried to take one sample from BRDF P3 with $W3 = p_3/(p_3+p_2)$, and one from P2 PDF with $W2 = p_2/(p_2+p_3)$. I am not clear how to evaluate the integrand for samples taken from P2 distributions. This is the pseudocode:

for each measurement point x {
    while(1){
        //P1
        //DirectLight routine uses P1 if x is indoor hit or
        //sampling hemisphere if x is outdoor, to get direct light from sky
        totalEstimate += DirectLight(x) * throughput

        //P2
        y = Trace one sample from P2
        if y is external surface {
           weight = p2(y) / (p2(y)+p3(y))
           totalestimate += DirectRadiance(y) * weight * brdf(x) * dot(x, Nx) / P2
        }

        //P3
        z = Trace one sample form P3
        if z is a hit{
            weight = p3(z) / (p3(z)+p2(z))
            throughput *= weight * brdf(x) * dot(x, Nx) / P3
        }
        else
            break;

        //using russian roulette to terminate the path..
    }
}

I tried this but had increase in variance. The measurement points are few points in room floor area and a contour line is drawn based on a the values. The above gives higher variance than using only P1 and P3 with no MIS weights.

Update:

This excersise in PBRT seems to be relevant to this question. So I post it here
in case it helps.

I will not answer your questions directly, but will try to change your mindset a bit. I hope got your intentions reasonably well...

The usual way the path tracing is implemented is that the reflection integral is split into direct and indirect illumination and they are estimated separately.

The direct illumination can be estimated using more sampling strategies which are combined together using the multi-sample multiple importance sampling method by Veach. Typically, there is just BSDF sampling and light sampling used, but you can also add things like light portals sampling and other more sophisticated stuff. I guess this is where you are headed.

The indirect illumination is usually estimated by just one sampling strategy (typically BSDF sampling) because you want to avoid exponential growth of recursive paths count. That's the reason why path tracing has poor performance for indirect illumination reflected/refracted through glossy/specular materials.

A nice thing about the split integral is that you can share samples between the direct and indirect integral. If you have two integrals over the same domain and have a sampling strategy that can be used for both, then you can generate a sample to estimate the first integral and then use the same sample for the second integral as well. There is no problem with correlated samples here because we have two separate computations. In practice you generate a BSDF sample with some PDF, shoot a ray in that direction and use the intersection point to estimate both direct and indirect integral separately.

PS: I guess that for indirect illumination estimator you could possibly use the one-sample multiple importance sampling method by Veach to incorporate the light portals and other sampling strategies, but I have some doubts about the efficiency of the resulting estimator, especially when reusing the samples for the direct illumination integral.


Retrieve real distance from depth buffer for shadow ray (vulkan hybrid rendering) I'm trying to implement an hybrid renderer based on Vulkan extension VK_NV_ray_tracing. Basically, the rasterizer compute the albedo and depth map of the scene. Then I try to compute the distance from the depth buffer so that I can send a shadow ray from a raygen program to compute direct lighting. I linearize the depth to obtain a correct distance value from the depth buffer. What I am doing is this:

const vec2 texcoord = vec2(gl_LaunchIDNV) / vec2(gl_LaunchSizeNV);
const double z_b = texture(depthBuffer, texcoord).x;
const double depth = (2.0 * near * far) / (far + near - z_b * (far - near));

I am then computing the ray position and direction

const vec2 pixelCenter = vec2(gl_LaunchIDNV.xy) + vec2(0.5);
const vec2 inUV = pixelCenter / vec2(gl_LaunchSizeNV.xy);
const vec2 d = inUV * 2.0 - 1.0;
const vec4 camFront  = cam.viewInverse * vec4(0, 0, 1, 0);
const vec4 camPos    = cam.viewInverse * vec4(0, 0, 0, 1);
const vec4 camTarget = cam.projInverse * vec4(d.x, d.y, 1, 1);
const vec4 camDir    = cam.viewInverse * vec4(normalize(camTarget.xyz), 0);

And the final ray position is then set using the depth I just computed.

const vec3 pos = camPos.xyz + camDir.xyz * float(depth);

The problem here is that the depth seems incorrect because shadows tend to curve at the edges of the screen and do not fit the geometry.

I think this might be because of the depth buffer value that does not represent the distance from the origin of the camera but from the near plane, which would explain the problem, but I am not sure of that.

Any idea?

Thanks for your help, I fixed it by changing the way i am computing the depth :)

const double z_b = texture(depthBuffer, texcoord).x;
const vec4 clipSpacePosition = vec4(texcoord * 2.0 - 1.0, z_b, 1.0);
vec4 viewSpacePosition = cam.projInverse * clipSpacePosition;
viewSpacePosition /= viewSpacePosition.w;
float depth = length(viewSpacePosition.xyz);


How does this app pull off fast, high radius HDR I am in awe at this app I just found. It was actually one of the first things I installed when I first bought my iPad 2 but now they have updated it to add an HDR effect.

I have never seen a real-time glow effect that looked quite this amazing. I have no clue how the developer pulled off such a large radius in realtime graphics.

I have a game with similar unprocessed graphics to his. I have been trying to get a glow effect like this working but it has been near impossible. I am currently using separable convolution gaussian blur methods in addition to downscaling my blur mask. Still, it barely runs in real-time. It is nowhere near as big as a radius as this app pulls off and it certainly isn't as good of quality.

Is there some method I am not thinking of?

I am open to the possibility (but doubtful) that they are using really large textures with a falloff distance or that lighting is computed from the distance from a fragment to light sources. If you play with it and notice when the FPS drops it it just doesn't seem to have the right FPS to particle amount and HDR on/off quality to it to be that.

At CEDEC and GDC in the early 2000s, Masaki Kawase has presented a series of fast post-processing based lens effects, including large bloom.

This presentation from GDC 2003, Frame Buffer Postprocessing Effects in DOUBLE-S.T.E.A.L (Wreckless) (see slides 15 and upwards: "Bloom"), gives a first version of the bloom. It consists in doing a Gaussian blur by sampling at exponentially increasing distances.

In this presentation from GDC 2004, Practical Implementation of High Dynamic Range Rendering (see slides 44 and upward: "Glare generation"), he updates the technique. Instead of varying the sampling distance, he uses downsized versions of the original image, using a carefully crafted equation to achieve a large yet spiky Gaussian blur.


Mirror Reflections: Ray Tracing or Rasterisation? Are mirror-like reflections in computer graphics purely handled with ray-tracing/ray-casting techniques or are there some situations where they are achieved through rasterisation?

There are a couple of special cases where mirror-like reflections can be rendered efficiently using rasterization techniques, and these are commonly used in games, although they don't work for the general case.

Planar reflections

If the reflecting surface is flat or reasonably close to flat, the reflected image can be rasterized in an separate rendering pass, by applying a reflection transform and clip plane to the scene. The reflection is then composited into the final image using a shader (or, in older games, stencil buffer masking). Also, a small amount of distortion or blurring might be applied to the image if the reflecting surface isn't intended to appear perfectly flat and smooth.

This is very commonly used in games for water reflection in lakes or the ocean, as long as any waves aren't too large. This page from the UE4 docs shows a number of examples of planar reflections used for water. The same technique is also often used for mirror reflections in glossy floors, or for actual mirrors.

The main strength of planar reflection is that it automatically gives you correct perspective and occlusion in the reflected image. The main limitation, aside from being limited to planar surfaces, is the performance cost of rendering an extra copy of the scene (with full lighting, etc) for each reflection plane. For instance, having multiple water surfaces at different elevations (e.g. in a boat lock) would require another rendering pass for each elevation. For this reason, usually when planar reflections are used, they're set up in such a way that only one reflection plane is visible at a time.

Cubemap reflections

Another special case is when the reflecting object is reasonably small and, ideally, not too close to other objects around it. In that case, you can approximate reflections by rasterizing a cubemap around the object. Then it can be used as an environment map during shading, by sampling the cubemap along the reflection vector calculated in the shader.

Many games use pre-rendered cubemaps to render static environment reflections on characters and dynamic objects; somewhat less commonly, you can also dynamically re-render a cubemap every frame to get dynamic reflections of other objects. This might be used on the cars in a racing game, for instance.

The advantage of cubemap reflections is that they can be applied to any shape of object, not just planar ones. However, since the cubemap is rendered from a single point, the reflections have increasingly incorrect perspective and occlusion the farther you get from that point. This can be partly addressed by parallax correction, and by blending between multiple cubemaps based on location, but ultimately cubemaps just don't give you the accuracy of either planar reflections or raytracing. Moreover, re-rendering a dynamic cubemap every frame is quite expensive, as it requires 6 extra rendering passes (one for each face of the cube).

Hybrid techniques

There are also cases where rasterization is used to build an initial data structure for ray tracing / casting to traverse into.

Screen-space reflection can be understood this way, as it involves first rendering the screen using ordinary rasterization techniques, then treating the Z-buffer as a heightfield and marching rays through it.
Voxel cone tracing is a similar case, where the scene is voxelized using rasterization, and then rays can be marched through the voxel grid. This perhaps doesn't qualify as "mirror-like" reflections since in practice the voxel grid isn't high-resolution enough for that, but it could be done in principle.

So yes, depending on what degree of approximation you can accept and what constraints you can work within, there are quite a few cases where mirror-like reflections can be achieved using rasterization, or hybrid rasterization/raytracing techniques.


Normal 2D photo to VR-compatible spherical photo How to turn a normal photo like this one :

into a photo that I can use in my game development platform (Unity3D). I just wrap a 3d sphere model with the photo and then it can be all around me when I use it in a VR headset.

I know it's impossible for a car photo, but i just need to convert the space photo into the something like the car photo then retouch the stitching.

There are a few ways you can do it. Basically, you need to get your photo into a format that's suitable for using as an environment map. There are a number of such projections available. The most common ones I've run into for 360° video are:

Cube maps - 6 images that show what the camera would see if it were looking forward, backward, left, right, up and down from the same position.
Equirectangular - the 2nd image you show above is this type of photo. It must be exactly twice as wide as it is tall. The rows and columns of pixels then correspond to the latitude and longitude of the sphere you map it onto.
Dual hemispherical - this is when you take a camera (real or virtual) and make an image with a 180° field of view. There are 2 - one for the front and one for the back.

Given the image you have, there are a few ways you could create something useful from it. You could crop it or extend it so that it has a 2x1 aspect ratio and use it as-is. You'll get a weird seam where the left and right meet, but if you are handy with photo editing tools, you can probably modify it to look reasonable. Note that this will give you the correct image for an equirect projection, but it may or may not look how you want. That purple band would never be visible all at once. It would wrap around the entire field of view, for example.

Another option is to make a cube map out of it. You could crop what you have into 3 equal squares for, say, front, left, and right. Then you'd need to generate the other 3 yourself. Generating a star field isn't too difficult either in code or using a photo-editing/compositing tool. Or if you know any artists, perhaps they could help you.


Colour Depth in Hardware I hope this is the correct place to ask this question. It feels to me that I should ask people who have had to program graphics as there’s a lot of confusion about color depth in general.
I’m sure a lot of people are aware that Adobe Photoshop can render OpenGL to 10bpc/30bit displays as long as the graphics card, divers and monitor support it.
My question is, now that we have monitors that can display HDR, are these HDR monitors also capable of doing rendering in Photoshop 10-bit?

Maybe. HDR monitors are typically going to be some approximation of the DCI-P3 color space, while Adobe's wide-color rendering is the Adobe RGB color space. The two are similar but not quite the same; in particular, Adobe RGB extends a bit further into the green-cyan range than DCI-P3 does:

That said, these are only idealized color spaces, and the gamut of a given actual monitor is going to differ somewhat from these. That's why you'll see monitors advertised as having "99% Adobe RGB coverage" or "99% DCI-P3" or some such.
So, a typical HDR monitor can probably reproduce most of the Adobe space but maybe not quite all of it, but in practice it might be just about as good as an actual Adobe-supporting monitor.
That said, I don't know whether Photoshop and the drivers/OS/etc will actually cooperate to let you turn on Adobe RGB rendering on a DCI-P3 HDR monitor. If Photoshop is outputting 10-bit color in Adobe RGB space, but the monitor doesn't actually take that format, then it would need to be converted somewhere along the line to a color space and encoding that the monitor can take as input. That can certainly be done by the graphics card/drivers/OS, but I don't know whether the support for doing such conversions is actually there.


How does this small fractal shader work? I would like to know the magic behind this tiny fractal shader; I didn't really understand the code and the mentioned thread...

Is there an accessible explaination?

https://www.shadertoy.com/view/4ttBWM

Here is a line by line breakdown of my shader:

#define mainImage(o,p)\
    for( vec2 r = iResolution.xy, q = 2.*(p+p-r)/r.y; o.a++ < 30.; )\
        o += length( q = abs(q)/dot(q,q) - iMouse.xy/r) / 2e2

First, it is important to know that the “\” at the end of the first two lines do not count towards the character count and merely serve to make the code more readable by splitting it onto multiple lines.

#define mainImage(o,p)

Begins the definition of the function which would normally be written
void mainImage(out fragColor, in fragCoord){...}
This saves a few characters.

The first part of the loop:

vec2 r = iResolution.xy, q = 2.*(p+p-r)/r.y;

Establishes two variables of type Vector2:

r is the resolution which allows me to refer to iResolution several times later in fewer characters.

q is the coordinate in the fractal which is calculated from the fragCoord p and is then iterated in each loop.

Instead of fragCoord starting at (0,0) in the bottom left corner and extending to the (r.x,r.y) in the top right, (p+p-r) transforms each fragCoord so that the center of the screen is (0,0). This doubles the fragCoord value and then subtracts the resolution of the window.
The division by r.y fits the coordinates such that the top has a y value of 1 and the bottom has a y value of -1. With this, the x is scaled accordingly without stretching.
Finally, the multiplication by 2 just zooms out by a factor of 2.

Then for the iterator, I use o.a (the alpha channel of o).
o is the output color which is a vec4. As defined by Shadertoy, o begins with the value vec4(0,0,0,1). I use this so that I do not need to use characters declaring a new float as an iterator. o.a++<30 combines the test expression and the iterator in the for loop.

The content of the loop is one line where a number gets added to each channel of the output color. This line:

o += length( q = abs(q)/dot(q,q) - iMouse.xy/r) / 2e2

Can be broken down into a few parts.
Each iteration of the coordinates in the fractal occurs here:

q = abs(q)/dot(q,q) - iMouse.xy/r

Where iMouse.xy/r is a way to transform mouse coordinates from pixel to the range (0,0) to (1,1)

Each iteration of this function changes the value of q to be used.

What produces this fractal is the idea that some initial values for q will escape to infinity and others will converge or cycle around a few small values. Each iteration I increase the lightness of the pixel based how far q is from (0,0). This length of q is then divided by the constant 2e2 (200) which is to prevent o.rgb from increasing too rapidly.

The contents of the loop could be understood as two steps:

q = abs(q)/dot(q,q) - iMouse.xy/r;
o += length(q) / 2e2;

In the end, even if o contains a value in one of its channels that is greater than 1, it gets clamped by Shadertoy automatically, and a trailing semi-colon is not needed on the last line.


What is the difference and/or relationship between the OpenGL SDK and the OpenGL GPU driver? I'm brand new to OpenGL and graphics programming in general, and I'm trying to figure out what software I need to program a GPU.  I saw an NVidia OpenGL driver, and I've also seen reference to the OpenGL SDK.  My understanding (sort of) is that the OpenGL driver is essentially an API that hides the differences between different GPUs and the OpenGL API works on top of that.  I really have no idea though.

I wouldn't really call it OpenGL SDK but OpenGL API. There's a common OpenGL API which is basically a set of functions you can call to control the GPU. This API is hardware agnostic (barring extensions), so you can use it on NVIDIA, ATI, etc. GPU's.

What the OpenGL driver does is to turn these platform agnostic function calls to hardware specific commands that the driver then sends to the GPU. For example when you tell GPU to draw a triangle via OpenGL API, then the driver converts these calls under the hood to specific commands for the specific GPU you are using. This abstraction is important so that you don't have to write different code for every GPU.

To explain things a bit in lower level, the graphics API implementation operates in user mode, while the graphics driver operates in kernel mode. User mode is "protected" mode your program runs in, in the sense that you can't mess up your entire device but only your own app and it's also relatively fast to perform the API function calls in this mode. Kernel mode on the other hand is "unprotected" and performing kernel mode calls requires an expensive mode switch. For OpenGL the IHV's provide both the OpenGL API and driver implementations and thus IHV's can provide optimal switching between the modes. This is in contrast to DirectX where API implementation is provided by Microsoft and the driver by IHV's.


Is it possible to decode and play e.g. 60fps video as 30fps, if computer is too slow for 60fps? I am wondering this, as e.g. YouTube offers some videos 60fps, but computer may be too slow to play them.

It is likely somewhat possible, depending on how the video was encoded. The player can easily skip displaying half the frames but that won't gain us much. What needs to happen is to skip decoding some of the frames.

Modern video codecs nearly all use a mix of I, P and B frames. Most are quite flexible in which frame types are used and only specify what a decoder should be able to handle, leaving the encoder free to pick the most appropriate way to encode a video.

Because of this, even if we want to show only half the frames, we may still end up having to decode more than that. It is possible to build a video stream where every other frame can be skipped because it is not referenced by other frames. But I suspect that most encoders will prioritize encoding efficiency over this rigid requirement. So in practice, we will only be able to skip decoding a smaller number of frames, unless the video was specifically encoded to support this scheme.

The other issue is player support. The player would need to recognize that computing power is insufficient and specifically target half the frame rate. In practice, most that I've seen will skip erratically instead. They expect the lack of computing power to be temporary as that is the most common scenario.

The expected fix for a permanent lack of computing power is to use a lower quality video stream. For devices with well known computing power (eg. phones, tablets, etc), this can be handled automatically by not presenting the high quality alternative. I don't know specifically if YouTube's "auto" quality setting handles this the same way it handles insufficient bandwidth but it could certainly be done and would be a much better solution.

TL;DR: It's possible but usually not worth doing.


Correcting my "look-at" matrix so that it works on non-camera objects? I'm familiar with the "look-at" matrix and the goal of being able to rotate a Camera at a specific location towards a specified target.

When I tell the Camera to look at some target, there're no issues. However, when some other non-Camera object needs to "look-at" another object, such as a hypothetical Dog looking at some Food, so that the Dog "points" in its direction, it doesn't work as expected. In this case, when the look-at matrix used for the Camera is used (without modification) for the Dog, the Dog ends up looking in the opposite direction I intended it to face.

What follows is the code I've been using to build my look-at matrix:

public static Matrix4 createLookAtMatrix(
    Vector4 eyePosition,
    Vector4 targetPosition,
    Vector4 upDirection)
{
    // forward, side, up
    //
    // f = normalize(targetPosition - eyePosition);
    // s = normalize(f × normalize(upDirection));
    // u = s × f;
    Vector4 f = targetPosition.sub(eyePosition).normalize();
    Vector4 s = f.cross(upDirection.normalize()).normalize();
    Vector4 u = s.cross(f);

    float[][] values = new float[][] {
        { s.x(),  s.y(),  s.z(), -s.dot(eyePosition) }, // row 0
        { u.x(),  u.y(),  u.z(), -u.dot(eyePosition) }, // row 1
        {-f.x(), -f.y(), -f.z(), -f.dot(eyePosition) }, // row 2
        { 0f   ,  0f   ,  0f   ,  1f                 }  // row 3
    };

    return createFrom(values);
}

What am I missing? Is there a way to build such a "look-at" matrix so that it works for both regardless of who/what the viewer is (i.e. Camera or Dog)?

I'd like to avoid depending on "special corner cases" to detect what kind of object is trying to "look-at" something else, and so on.

My system can have multiple SceneObjects at any given moment connected to the same SceneNode (SceneNodes are the objects that actually contain the transforms) and I really need a way to build a "look-at" matrix that will work regardless of which kind of object happens to get attached to the SceneNode.

This is a screenshot of the reference book I was reading/using when writing my code above:

You don't seem to get the concept right:

A look-at camera matrix is created by defining two points, the "from" point (the dog eyes for example assuming you take the position of the eyes of the dog and average them to get the point in between) and a "to" point which is where the dog looks at. So if your dog is at (3,3,3) and that you want it to look at the some food which is at (-10, 100, 20), then you need to call the lookAt() method with these 2 points and it will create a matrix that will actually orient and position the camera properly.

Now you are getting confused (probably) because most cameras in most 3D packages are positioned at the origin (by default) and looking down the z-axis (by default). You can create such camera with a lookAt() function btw that would look like that:

lookAt(Vec3f(0), Vec3f(0,0,-1));

For example. Now check what the matrix this call will produce with the matrix of the default camera in Maya. You will notice they are different. The second is an identity matrix, the first one is not.

Now for example if you create a default camera in Maya it will be position at (0,0,0) and looking down the negative z-axis. If you look at the 4x4 matrix of this camera you will get identity. And maybe what you are trying to do is to create a 4x4matrix using the lookAt function and multiply this matrix by the Maya's default camera matrix in which case, your camera will point in the wrong direction (it will be flipped along the camera's local coordinate z-axis). That's because the matrix returns by lookAt properly orient the camera (as suggested in the first paragraph) but since the default camera in Maya is already oriented along the negative-z axis, your Maya camera will look the wrong way.

So in summary:

If you use the lookAt method in a renderer, it will work as you expect. There's no z problem.
if you try to use it on a camera that is already oriented along the negative z-axis and yet has identity matrix yes indeed you will be looking the wrong way. This can be fixed easily by flipping the camera back one more time along the z-axis. Simply create another matrix that scales the object/camera by (1,1,-1).

EDIT

The OP says (see comment):

  I did spot what looks like a typo: it says "(mind the direction of this vector: it is To − From not From − To)", but then you go on to write the line of code Vec3f forward = Normalize(from - to);, which contradicts what you were trying to clarify.

But a note in the reference says:

  The camera is aligned along the negative z-axis. Does it mean I need to rotate the camera by 180 degrees along the y-axis or scale it up by -1 along the z-axis? Not at all. Transforming a camera is no different from transforming any other object in a scene. Keep in mind that in ray-tracing, we build the primary rays as if the camera was located in its default position. This is explained in the lesson Ray Tracing: Generating Camera Rays. This is when we actually reverse the direction of the rays. In other words, the z-coordinates of the rays' directions at this point are always negative: the camera in its default position, looks down along the negative z-axis. Those primary rays are then transformed by the camera-to-world matrix. Therefore, there is no need to account for the default orientation of the camera when the 4x4 camera-to-world matrix is built.

So if you look at the reference they use by default a right-hand coordinate system, which means x-axis points to the right, y is up and z goes out of the screen (outward, going your direction when x-axis points to the right). So they build a matrix that "follows this convention" which makes sense. Then they say in the note "yes but wait, normally the camera points down the negative z-axis, so you should be doing to - from and not from - to, and they explain that in the case of ray-tracing the fact that the camera looks down the z-axis is performed when the rays are cast (by inverting their z-coordinate).

Vec3f rayDirectionInCamSpace{ x, y, -1 }; // invert here
normalize(rayDirectionInCamSpace);
Vec3f rayDirInWorldSpace = transform(rayDirectionInCamSpace, camToWorldMat);
pixelColor = trace(rayOrigWorldSpace, rayDirInWorldSpace);

So no it's not a typo. Now the reference you are pointing out maybe assume this matrix needs to be used in some OpenGL render in which there is no possibility to invert they ray direction when they cast. So they indeed use to - from instead which is the same as multiplying the cam-to-world matrix you would get from the reference by a matrix that flips the coordinates z-axis (multiply by (1,1,1-)).

So no typo, things are quite logical in both cases. The result you get is always constrained to a lot of small details, such as the handiness of the coordinate system and how is the matrix later used down your graphics pipeline (OpenGL pipeline, ray-tracer, renderer, animation package, etc. which may follow its own conventions, etc.).

EDIT 2

Note that in the reference your provide (screenshot of book) they do target - eye but then they use the negative of that in the matrix construction. Not sure why. Then also no idea why the multiply the eye by the vectors for the translation part of the matrix. It seems to be using row-order matrices for the axis of the coordinate system but then the translation part should be at mat[3][0], mat[3][1] and mat[3][2] and not mat[0][3], mat[1][3] and mat[2][3]. Not sure what book this is but seems to be terribly wrong and confusing.

EDIT 3

Code (that seems to be the only solution):

void lookAt(const Vec3f& from, const Vec3f& to, Matrix44f &V)
{
    static const Vec3f tmp(0, 1, 0);
    Vec3f forward = normalize(from - to);
    Vec3f right = crossProduct(normalize(tmp), forward);
    Vec3f up = crossProduct(forward, right);

    V[0][0] = right.x;
    V[0][1] = right.y;
    V[0][2] = right.z;
    V[1][0] = up.x;
    V[1][1] = up.y;
    V[1][2] = up.z;
    V[2][0] = forward.x;
    V[2][1] = forward.y;
    V[2][2] = forward.z;

    V[3][0] = from.x;
    V[3][1] = from.y;
    V[3][2] = from.z;
}

int main(int argc, char **argv)
{
    ...
    lookAt(from, to, camToWorldMatrix);
    worldToCamMatrix = camera.camToWorldMatrix.inverse();

    Matrix44f projectionMatrix;

    Matrix44f openGLWorldToCamMatrix = worldToCamMatrix;
    Matrix44f openGLProjectionMatrix = projectionMatrix;

    while (!glfwWindowShouldClose(window)) {
        glViewport(0, 0, width, height);

        glUniformMatrix4fv(projectionLocation, 1, GL_FALSE, (const GLfloat *)&openGLProjectionMatrix[0][0]);
        glUniformMatrix4fv(viewLocation, 1, GL_FALSE, (const GLfloat *)&openGLWorldToCamMatrix[0][0]);

        ...
        wwapBuffers(window);

    }

    return 0;
}


How to draw surface normals from surface normal maps? Using deep learning to estimate surface normals from monocular RGB images is a common task. The resulting image generally looks like this:

My question is how can I use this map to actually draw a normal vector (arrow) for any given point on the image? For example, I want to draw a line on the countertop that points up.
Most relevant tidbit I found on the topic: "The RGB color channels (red, green, and blue) in a normal map correspond to the respective X, Y, and Z coordinates of surface normals."
But I don't get what this means. The values are standard RGB, i.e., for each coordinate, you get three numbers in range [0, 255]. Just by how color changes between surfaces, it is clearly related to normals (obviously) but when I tried quiver plots using matlab I couldn't get reasonable results.
TLDR: What is the relation between surface normal maps and actual normal vectors? How can I draw a 3d line given a point on the image which represents the normal vector perpendicular to that surface at that point?

Typically the normal vector, which has XYZ components in [−1, 1] is linearly mapped to the RGB range [0, 255]. You can retrieve the normal vector by reversing this mapping.
For completeness, the encoding is:
$$
r = \text{round}((0.5x + 0.5) \cdot 255) \\
g = \text{round}((0.5y + 0.5) \cdot 255) \\
b = \text{round}((0.5z + 0.5) \cdot 255)
$$
Decoding is:
$$
x = (r/255) \cdot 2 - 1 \\
y = (g/255) \cdot 2 - 1 \\
z = (b/255) \cdot 2 - 1
$$


What is this technique called and how can I reproduce it algorithmically? I don't know the name of this technique, but the idea is to adjust the density and location of identical dots or small objects in order to produce a gray tone image or even a suggestion of three dimensionality. It's better to use these two images below as examples.

I'd like to do something like this on my own, but by writing relatively simple algorithm rather than a program or package. I'll be starting with a 3D representation of an object's surface inside a python script running in  Blender. My question isn't how to implement, but how to understand what's happening.

Could someone explain what this is called and share some amount of insight into how this works? I don't need a full-blown derivation, just some idea to get started on something simple.

In the screenshot, the objects (in software) follow the face contour in 3D, but their position (and brightness) is fluid and their separation varies continuously. While they appear to be approximately on a grid, I'm not sure if they are a fixed projected grid (I think I've read somewhere it was done using a Kinect) and move because of the movement of the model, or if they move algorithmically as well.

edit: More info on the Kinect in the Below the Line article Level 256 VFX Produces The Final Moments of Karl Brant for Nerdist.com.

  “Leveraging the power of open source applications like Processing, Python, and bash scripting, we were able to take the point cloud data from the Kinect, manipulate it, and get it into Nuke for integration with the live-action plates for texture re-projection,” Kleinberg explained. “The process evolved and grew over time, and when we finally saw the fruits of our labor on the screen, we were excited to see Karl Brant fully realized, taking on an expressive life of his own.”

I'm familliar with something simple like using a random number to decide if the pixel should be on or off depending on the gray value; if it's 0.3, then if a random number (chosen for each pixel, but not necessarily randomized every frame) is below 0.3, the pixel should be ON (seems to be called 1-bit dithering). But I think there is a lot more going on here than that.

below: Cropped screen shot at 11:23 in the short film The Final Moments of Karl Brandt:

I've done something similar. The way I did it was by using a height map image to change the positions of vertices in a grid in a vertex shader. I generated a 2D grid of vertices that were evenly spaced on the CPU. I uploaded the vertices and a texture and in the vertex shader, I sampled the texture and used the resulting brightness as the height. So I'd keep the x and y positions of each vertex and change the z based on the image.

Now there are other ways you could utilize this to get something more like in the video. You could either have a 2D grid of single vertices, or even a grid of small 3D models. Change the z position based on the color of a height map. But additionally, you could upload a small noise texture and vary the x and y positions by a small amount based on the noise, which might vary per frame. I would see if you could vary the x and y by a large enough amount to see, but small enough that no point ever crossed over a neighbor point. That would get you some more movement and randomness.


Project quad onto ellipse in 3D I have three points P0, P1, P2, which are located on an arbitrarily oriented ellipse in 3D space. I have a square texture map with a circle on it. I would like to render a textured quad (with the obvious UV mapping) so that the circle in the texture map projects onto the ellipse that P0, P1, P2 are on.

How do I calculate the vertices of the quad to project the ellipse correctly?

(Alternately, in addition to P0, P1, P2, I have the Keplerian elements of an elliptical orbit, if that's more straightforward.)

I assume that your texture is deemed to belong to the plane of the trajectory.

Looking at the unprojected ellipse, you can map the circle to it by an anisotropic scaling. If necessary, rotate the texture first if there is any orientation requirement.

To achieve this first mapping, you will compute the four corners of a rectangle, presumably related to the frame formed by the ellipse axis.

Then transform those vertices to the global scene coordinates, the same way to transform the ellipse.


Vulkan best way of updating pipeline viewport I have discovered that there are two ways to update the viewport (might be more, but not sure). Currently I am rebuilding the Pipeline when I resize my screen but there is a noticeable stutter as it rebuilds, but I have also found out that you can use vkCmdSetViewport when building/rebuilding your command buffers. But is this the right way to do it? And if I used vkCmdSetViewport, what should be put into the VkGraphicsPipelineCreateInfo pViewportState?

If there is an even better way of handling screen resizing please let me know.

Both suggested methods are fine, but they should be preferred in certain situations.

Dynamic state should be preferred in situations where you have many, perhaps thousands of, pipeline objects or you need to change the resolution often (such as in some sort of windowed tool. The downside to this is that it may reduce runtime performance since the driver may not be able apply optimizations for this.

Having the viewport only defined when building the pipeline objects might be more appropriate for games or simulators where you don't change the resolution often. Of course, when the viewport does change, it hits performance hard during resizing. This also has the drawback of needing to generate new pipeline objects, which could be especially problematic if you cache pipelines in a hash table of something similar.


Confused with PBR implementations I'm discovering the wonderful world of math and I'm hitting a huge wall with PBR and naming so I have a few questions:
As far as I understand BRDF describe how a material react to light (incoming direct light and reflections). But in "Cook-Torrance" I don't see anything about reflections I don't understand how I integrate it so I only get diffuse light and reflections.

I've read a lot about different "diffuse term" like "Oren-Nayar" is it a part of Cook-Torrance like a different "D" in the DFG nominator ? does it replace it without specular calculations ?

I've read a lot about "GGX" from my blender artist background I thought it was something that make a blurry reflection but It seems I was wrong and now I find this GGX term in a lot of different situations like in micro-facet normal distribution in this tutorial

So in short the question is: how "Oren-Nayar" and "GGX" and "Cook-Torrance" are related to each other (if they are) and how do I integrate "vec3 reflection" in those algorithm ?

As I said before I'm a newbie in math so If your answer is "math heavy" please write a glsl/hlsl version of your equations it's much easier to understand for me so an equivalence will help me a lot

PS: I understand my question is blurry but it is also as blurry as my understanding of the subject.

DFG pops up in the family of microfacet based BRDFs. It is simply the product of three terms:

D : The microfacet distribution.
F : The fresnel coefficient.
G : The geometric attenuation between microfacets.

When someone says Cook-Torrance, they usually mean a microfacet BRDF where the distribution (D) is Beckmann, which I think is what the original Cook-Torrance paper was about.

GGX is really just a different distribution (D term). GTR is another one. The names are often used as a shortcut to mean a BRDF using that distribution.

Now all of these assume specular reflection (or refraction) on the microfacets, hence the fresnel term. If you take a similar idea but use diffuse (lambertian) microfacets, you get the Oren-Nayar BRDF. I think that one is based on a gaussian distribution of the microfacets but I'm not familiar with the details.


Spherical Harmonics Diffuse CubeMap. How to get coefficients? envmap.pdf | An Efficient Representation for Irradiance Environment Maps
Spherical Harmonics and applications in real time graphics

I am using Spherical Harmonics to turn reflective(mirror like) cube map to diffuse light cube map.

From what I understand for an image, I calculate 9 coefficient(for red green blue) that describes the color of the cube map.

The nine coefficients are this...

And when you visualize them they become...

Then I do something with these coefficients to get color of the cube map.

The part that I am not understanding is using an image to get that 9 coefficients. All the papers kinda assume that you somehow got that coefficient calculated because it is "easy".

I don't know how to do that calculation.

Update

So while I were looking at Spherical Harmonic Lighting | Simon's Tech Blog

I found this equation...

So now I know that the mysterious yellow part of the equation was "solid angle related". I do know how to do solid angle but I still don't understand how to do it.

So at start you have samples from your cube map. Each sample has color and normal (dir) at which you sampled that color.
This is how I do it. I use coeffs from this paper (the same you linked), there are values for the first 9 of them.

So for constructing for each sample you:

Compute your SH basis using normal from sample

float Y00     = 0.282095;
float Y11     = 0.488603 * normal.x;
float Y10     = 0.488603 * normal.z;
float Y1_1    = 0.488603 * normal.y;
float Y21     = 1.092548 * normal.x*normal.z;
float Y2_1    = 1.092548 * normal.y*normal.z;
float Y2_2    = 1.092548 * normal.y*normal.x;
float Y20     = 0.946176 * normal.z * normal.z - 0.315392;
float Y22     = 0.546274 * (normal.x*normal.x - normal.y*normal.y);

Then multiply these coeffs by your color (L)

vec3 L00   = L * Y00;
vec3 L11   = L * Y11;
vec3 L10   = L * Y10;
vec3 L1_1  = L * Y1_1;
vec3 L21   = L * Y21;
vec3 L2_1  = L * Y2_1;
vec3 L2_2  = L * Y2_2;
vec3 L20   = L * Y20;
vec3 L22   = L * Y22;

And thats it, you have your coeffs. Now, you can store them in some data structure. You accumulate each of them like:

someArray[0] += L00;
someArray[1] += L11;
etc...

And you divide these accumulated coeffs by a number of samples;

Then when reconstructing :

compute your SH basis one more time for normal at point of the scene.
for approximating diffuse irradiance at given point of scene,
convolution with cosine lobe is performed using normal from that point.
Convolution means, for each SH order, multiplication by $
   \hat{A}=[\pi, \frac{2}{3}\pi, \frac{\pi}{4}]$

$\hat{A_0} = 3.141593$,
$\hat{A_1} = 2.094395$,
$\hat{A_2} = 0.785398$

L coeffs are sampled from data structure.

vec3 color = A0*Y00*L00
             + A1*Y1_1*L1_1 + A1*Y10*L10 + A1*Y11*L11
             + A2*Y2_2*L2_2 + A2*Y2_1*L2_1 + A2*Y20*L20 + A2*Y21*L21 + A2*Y22*L22;

Disclaimer: This basically works (for me) but I am not an expert and something might be missing or not perfect because this IS confusing topic...


preserving z-values during projection? Consider this figure for projection.

There are two equations that give value for the xp and yp coordinates:

$x_p=x.\frac{(z_{vp}-z_{prp})}{h}+ x_{prp}.\frac{(z-z_{vp})}{h}$ and
$y_p=y.\frac{(z_{vp}-z_{prp})}{h}+ y_{prp}.\frac{(z-z_{vp})}{h}$

and then it explains in a paragraph: "Setting up matrix elements for obtaining the homogeneous-coordinate $x_h$ and $y_h$ values in the given equations is straightforward, but we must also structure the matrix to preserve depth (z-value) information. Otherwise, the z coordinates are distorted by the homogeneous-division parameter h. We can do this by setting up the matrix elements for the z transformation so as to normalize the perspective-projection $z_p$ coordinates. There are various ways that we could choose the matrix elements to produce the homogeneous coordinates  and the normalized $z_p$ value for spatial position (x, y, z)."

So, I have some questions regarding this:

What does it mean to preserve depth(z-values)?
What does it mean to normalize the perspective-projection zp coordinates?
How to perform this normalization?
How does this normalization preserve z-values?

As you can see in the figure above, during a perspective projection, the goal is to essentially "splat" points within the view frustum onto the near frustum plane. That means that the final z coordinate is actually just fixed at $z_{vp}$.

Now, if we did this poorly, we could just map $z$ to $z_{vp}$ directly in our projection matrix by setting the third row of the matrix to $\left[0, 0, \frac{z_{vp}}{z_{prp}}, 0\right]$ and setting the final homogenous coordinate to $1$ (so no divide by $w$ happens at the end). This approach is problematic because squashing all this depth information to $z_{vp}$ loses information and we can no longer do things like a depth test.

This is why, instead of mapping all $z$ values to the same fixed distance from the camera, we opt to map it into a range of $z$ values instead. This is the "normalization" that the text is referring to. Furthermore, we aren't preserving the "values" at all. The wording in the text is confusing, but what we are preserving is the information about what points lie further away from the view origin. When it says that we have a choice about how to do this normalization, well, we can choose to map $z$ into any range that we wish, and in graphics contexts, we normally want something that preserves precision well enough that we can do nice things like get crisp shadows and such.

If you look at standard texts, the $z$ value will be mapped via the matrix row:

$$\left[ 0, 0, \frac{-(f + n)}{f - n}, \frac{-2fn}{f - n} \right]$$

and the $w$ coordinate is mapped to $-z_{prp}$. This should look similar to the other entries that map $x$ and $y$ between the right-left and top-bottom frustum planes respectively. In this case, we're mapping $z_{prp}$ between $n$ and $f$. Now, we have a $z$ coordinate that is mapped in a range such that after the divide-by-$w$ to get back to normal 3-coordinate space from homogeneous space, maps back to $z_{vp}$ as expected, but until that occurs, can be used to perform depth comparisons. The choice of $n$ and $f$ is arbitrary, but too narrow and you end up clipping $z$ values that lie outside the range, and too wide and you lose precision. Some people use a $-\log$ depth computation instead to gain as much precision as possible, but I digress.


What are the best ways to start graphics programming? [I got a warning that the question will be closed].

I have identified that CG is the place where all the advanced algorithms and computations are taking place.

What is the best way to start from rudimentary and progress gradually in CG programming?

Please express language choices (C, C++, etc.) and other topics you find relevant. I also found out  Michael Abrash's book.

There are many possible answers depending on your OS, goals and wanted level of abstraction. For low-level programming you can start by learning OpenGL, Metal or Direct3D. Most used shader languages are HLSL based. For higher level there are things like ShaderToy and engines like Unity (C#) or Unreal (C++) where you can experiment with graphics coding.


Lambertian Shading of a cube In theory I can't use a lambertian shader on a cube, because there aren't surface normals on the edges.

Is this correct ?

Why is this always possible in software applications ?

Each cube face is made up of a quad, or two triangles.  Let's take 3 points from either the quad, or one of the triangles, and call them $A,B,C$.

You can get the normal to this surface by normalizing the cross product of $\overline{AB}$ and $\overline{BC}$.  You can then use that surface normal to calculate lighting.

You can do similar calculations to get the tangent and bitangent to then calculate $u,v$ values for texture mapping.

I'm not sure why the edges not having normals makes you think you can't calculate lighting for the cube faces, but you totally can.

You could even use the vertices to calculate a normal for the edges if you wanted, but I'm not sure why you would want to do that :P


Convert a .X mesh to a modern format I have a bunch of meshes in the .X format, which isn't supported by many 3D applications today. Therefore, I would like to convert these to a modern format like fbx or gltf/glb.

Since these meshes are proprietary, an offline solution would be preferred. What would be an ideal way to achieve this lossless-ly?

Blender appears to have a .X importer.  It's a plugin that ships with Blender, but is disabled by default, so you have to enable it in User Preferences.  See: https://blender.stackexchange.com/a/8749/18161

Blender also has a glTF exporter.  For now that's a separate download, from here: glTF-Blender-Exporter