Dynamically allocating morph targets in shaders So with the idea that I want to load in an object with morph targets and for simplicity sake I know it will have between 1-8 position morph targets. Ultimately I want to do

renderedPrimitive.POSITION = primitive.POSITION +
   weights[0] * primitive.targets[0].POSITION +
   ....
   weights[N] * primitive.targets[N].POSITION;

But being open to load in any model how do you go about knowing the number of vertex input attributes to allocate. Getting the CPU side dynamically is not an issue, but am I wrong that the ONLY way around this is building your shader with some #define and compiling during runtime?

With modern hardware, morph targets probably should not be VS inputs. They should be SSBOs or the D3D equivalent. The vertex shader should fetch from these arrays of positions and act on them appropriately. You can pass the number of targets as a uniform or something.


What is the most physically accurate representation of color possible in computer graphics? I am looking for theoretical and implemented models to represent color in software that strives for physical accuracy.

I know that "color only exists in the mind" and that it sort of does not make sense to look for "physical accuracy", but still, how much better is spectrum sampling or other more elaborate representations of color to simple RGB, and what benefits will it bring? What drawback beside performance will it incur (or is performance a drawback)?

The most physically accurate way would be to have a $l(\theta)$ which for each possible color frequency has a certain value.

Converting to RGB would then need a frequency responce function for each channel and the result is then $\int_{infrared}^{ultraviolet} l(\theta)F_{red}(\theta)d\theta$.

Reflected light then has a 2 dimensional response function: $L_{reflected}(\theta) = \int l_{incoming}(\phi)R(\phi,\theta)d\phi$. (ignoring the BRDF here for simplicity)

Drawbacks are that instead of just 3 color channels you now have infinite channels to worry about.


Given a picture of a unit square at an arbitrary angle and position, how can I reconstruct the distance and orientation? To simplify, assume faces and corners of the square are labeled and the camera lens if whatever makes this simplest (a 90 degree field of view projection?)

The application: using a known and labeled unit square to assist an AR app.

For the case of a perfectly centered rotated on one axis, we can use the estimated distance from the axis the square was rotated around plus the ratio of sizes of the distorted sides.

But for an arbitrary view, the problem becomes much much harder.

Links to the general case of this problem in projective geometry also appreciated

This problem is known as camera calibration. The goal is estimating the various parameters of a "camera":

The intrinsic parameters: Things like focal length, distortion of the lens, sensor parameters etc.
Extrinsic parameters: Orientation of position of the camera.

To robustly determine those from an image that contains labelled points you usually need more than one image. A typical way to do this is recording many images of a chessboard pattern. But as soon as you have determined all the intrinsic parameters, you could save those for future applications as they do not have to be estimated over and over again.

I recommend reading the introduction to camera calibration of openCV:

https://docs.opencv.org/3.1.0/dc/dbb/tutorial_py_calibration.html

https://docs.opencv.org/3.1.0/d9/db7/tutorial_py_table_of_contents_calib3d.html


What actually happens in frame buffer when an image or document is scrolled? What actually happens inside computer when the text, pdf, jpeg, ms-word or any other document(file) is scrolled? Does it generate a new bitmap image sufficient for the screen to display even for tiny scroll or are does it show part of bit map image for each scroll? Please describe what is the basic idea behind the working of scroll in laymen terms?

In the olden days of text-mode terminals, the "frame buffer" was a buffer of text, not an image. Scrolling worked in the way you described. The buffer was larger than would fit on the screen, and to "scroll" it would change an offset in the display controller, so it would display different lines. This only makes sense when the whole framebuffer is scrolled at once, which never happens in a windowing system.

Nowadays, it depends on the application. A word processor will rasterise the text separately each frame: there's no off-screen buffer at all. It wouldn't make sense to store an image of the whole document, because the memory use would be huge, and the whole thing would have to be recreated every time the text changed.

On the other hand, an image editor will probably store the whole image, possibly as a texture in GPU memory. A program on the GPU copies the visible section of the image to the application's frame buffer, each frame. This can be done by simply having the view parameters (offset and zoom level) as uniforms and updating them as the view is changed. This is actually very fast even though it's essentially a copy, because GPUs are good at drawing textures efficiently.

A web browser might use either technique, or something more sophisticated. There was a trend in mobile browsers a couple of years ago to render in tiles. The structure of the page would be rasterised to a set of off-screen tiles: enough to more than cover the screen. A program on the GPU draws all the visible tiles into the correct place in the application's framebuffer, clipping the edges. That way, small scrolls or zooms can be performed by changing the offsets of the tiles, without having to rasterise the page again. When you scroll too close to the edge of the tiles, a new row of tiles can be rasterised in a separate thread.

This tile approach is fast to display, uses a fixed number of fixed-size textures, and allows a trade-off between responsiveness and scroll speed. It also allows lower-resolution tiles to be used if the page is too slow to rasterise during a fast scroll. It does require a lot more sophistication in the way you rasterise the page to the tiles, to know which parts of the page can affect which tiles.

Whichever technique the application uses for managing off-screen content, it's responsible itself for producing a new frame each time. The windowing system or compositor doesn't give it a way to specify a window of a larger texture to use without copying.


How can I intercept and filter all frames coming out of SteamVR? I'm trying to investigate the effect of certain image operations on how VR scenes are perceived. To do this, I'd like to run an off-the-shelf SteamVR application, capture the frames as they come out of the compositor, run a shader or compute kernel on them, and have them displayed as normal on the HMD. I'm aware of the latency penalty for doing this; I have some ways to minimise it, and in this case I can tolerate extra latency, so don't worry about that for now.

I can see three strategies which might let me achieve this, but I don't know enough about the display subsystem to know which will work and how to start.

Make a 'fake' display driver. My display driver would take the place of the HMD's driver, and it would act like a proxy. When the SteamVR compositor tries to swap buffers, my driver would do the processing on the backbuffer before poking the real HMD driver. It seems like the main difficulties in this would be (a) getting Windows to accept my fake driver, (b) getting SteamVR to accept my fake driver, and (c) letting my driver use the real HMD driver.
Use normal video capture software (such as OBS) in display capture mode to capture the frames before they get to the display driver, and then use an overlay technique (like the Steam overlay) to draw over the frame completely with my processed frame. This sounds simple, but I know there's a lot of work in making either of the two halves of the system.
Hook into the SteamVR compositor itself. It supports third-party HMD integrations, and I'd hoped I'd be able to trick it into using my shader as a chromatic aberration correction shader, but it turns out that it doesn't let the HMD integration run a shader directly, it just sets the parameters for the built-in shader. This seems like it should be the easiest way, if I can just find a way to make the compositor run my shader. (I'm not averse to hacking the compositor if there's a way.)

I know some people on the site are also active researchers in this area, so I'm hoping someone has practical experience of what worked and what didn't. Since this is for an experiment, not production software, I'm more interested in an answer I can hack up in a few days to work on one particular PC, than in something that's reliable but takes months to get right.

So far, I've been able to attach to vrcompositor.exe with RenderDoc and intercept the D3D calls that do the composition. According to Baldur Karlsson (the author of RenderDoc), it's not that hard to hack RenderDoc to use its hooks to run arbitrary code on each Present call of the chain I'm interested in.

I've not had a chance to try that out yet, so this isn't a final answer! I'll update this post with any further progress.


How to use Shaders with vertex & index buffers? I'm trying to render a randomly generated terrain with vertex buffers and OpenGL. I want to use a shader to color the terrain, but it seemms like my shader isn't used. I have used shaders before when drawing triangles manually on the CPU and didn't have any problems there.

activate and deactivate is just glUseProgram(m_ShaderProgram) and glUseProgram(0);

This is my draw method:

void Terrain::drawTest() {

    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, m_IndexBuffer);
    glBindBuffer(GL_ARRAY_BUFFER, m_VertexBuffer);

    glEnableClientState(GL_VERTEX_ARRAY);
    glEnableClientState(GL_NORMAL_ARRAY);

    // setup position & normal pointers
    glVertexPointer(3, GL_FLOAT, sizeof(TerrainVertex), BUFFER_OFFSET(0));
    glNormalPointer(GL_FLOAT, sizeof(TerrainVertex), BUFFER_OFFSET(12));

    m_ShaderProgram.activate();
    glDrawElements(GL_TRIANGLES, indicesCount, GL_UNSIGNED_INT, BUFFER_OFFSET(0));
    m_ShaderProgram.deactivate();

    glDisableClientState(GL_VERTEX_ARRAY);
    glDisableClientState(GL_NORMAL_ARRAY);
}

And my shaders look like this:

Vertex shader:

varying vec3 Normal;
varying vec3 Position;
varying vec2 Texcoord;

void main(void) {
    Position = (gl_ModelViewMatrix * gl_Vertex).xyz;
    Normal = gl_NormalMatrix * gl_Normal;
    Texcoord = gl_MultiTexCoord0.xy;
    gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
}

Fragment shader:

varying vec3 Normal;
varying vec3 Position;
varying vec2 Texcoord;

void main(void) {
   gl_FragColor = vec4(1.0,1.0,0,0);
}

Result:

As you can see the fragment shader doesn't seem to do anything and I can't figure out what the problem is.

Apparently there was just a problem with the loading of the shader. A bug that crept into the code from an old project...

EDIT: The GLuint of the compiled shader was never saved in the class and only local to the load function. Just a stupid mistake really.


How to compute A and B in projection matrix I'm trying to compute a projection matrix to transform from view space to NDC with a near clip plane at -1 and far plane at +1. The general form of this matrix (disregarding aspect ratio and focal length) should be

$\begin{bmatrix}1&0&0&0\\0&1&0&0\\0&0&A&B\\0&0&-1&0\end{bmatrix}$

I followed SongHo's guide at http://www.songho.ca/opengl/gl_projectionmatrix.html which sets $A=-(f+n)/(f-n)$ and $B=-2fn/(f-n)$.

However, setting the near clip plane at $n=-1$ and $f=-10$ (in view space) and using these $A$ and $B$, I get points with $z$ values on the interval $[-1, -10]$ transformed to $[3.04, 1.04]$ (after homogenisation).

When I do the derivations myself, I'd like to set $A=(n+f)/(n-f)$ and $B=(-2fn)/(n-f)$ instead, which indeed transforms to $[-1,1]$ instead.

Am I doing something wrong?

Note: I'll be using column vector notation (eg. $Ax=y$), thus if you are using row notation, transpose the matrix and multiply from the left instead of the right. (eg. $y=x^TA^T$)

A perspective transformation matrix transforms a view frustum into a rectangular view volume. Note that the resulting view volume might not be a cube and might not be centered at 0.

One example of a perspective transformation matrix. There are many more, and yours is included.

Let $l=left,\\ r = right,\\ b = bottom,\\ t = top,\\ n = near,\\ f = far$

$M_{persp}=\begin{bmatrix}
1& & & \\
 &1& & \\
 & &\frac{n+f}{n}&-f\\
 & &\frac{1}{n}&
\end{bmatrix}$

Since the resulting view volume is not the canonical view volume, we need to apply another matrix.

A orthographic transformation matrix transforms a rectangular view volume into a cubic view volume with two corners at (-1, -1, -1) and (1, 1, 1), which happens to be the canonical view volume.

Intuitively it will need one translation and one scaling.

$\text{T}_{ortho} = S\left(\frac{2}{r-l},\frac{2}{t-b},\frac{2}{n-f}\right)T\left(\frac{-(l+r)}{2},\frac{-(b+t)}{2}\frac{-(n+f)}{2}\right)$

$M_{ortho} =
\begin{bmatrix}
    \frac{2}{r-l}& & & \\
    & \frac{2}{t-b} & & \\
    & & \frac{2}{n-f} & \\
    & & & 1
\end{bmatrix}
\begin{bmatrix}
    1 & & & \frac{-(l+r)}{2} \\
    & 1 & & \frac{-(b+t)}{2} \\
    & & 1 & \frac{-(n+f)}{2} \\
    & & & 1
\end{bmatrix}
$

$M_{ortho} =
\begin{bmatrix}
    \frac{2}{r-l}& & & -\frac{l+r}{r-l} \\
    & \frac{2}{t-b} & & -\frac{t+b}{t-b} \\
    & & \frac{2}{n-f} & -\frac{n+f}{n-f}\\
    & & & 1
\end{bmatrix}
$

Now you can combine them together to form a perspective projection matrix.

$M_{proj} = M_{ortho}M_{persp}$

$M_{proj} =
\begin{bmatrix}
    \frac{2}{r-l}& & & -\frac{l+r}{r-l} \\
    & \frac{2}{t-b} & & -\frac{t+b}{t-b} \\
    & & \frac{2}{n-f} & -\frac{n+f}{n-f}\\
    & & & 1
\end{bmatrix}
\begin{bmatrix}
1& & & \\
 &1& & \\
 & &\frac{n+f}{n}&-f\\
 & &\frac{1}{n}&
\end{bmatrix}$

$M_{proj} =
\begin{bmatrix}
    \frac{2n}{r-l}& & \frac{l+r}{l-r} & \\
    & \frac{2n}{t-b} & \frac{b+t}{b-t} & \\
    & & \frac{n+f}{n-f} & \frac{2fn}{f-n}\\
    & & 1 &
\end{bmatrix}$

If you have a free-moving camera you will need a camera matrix too, thus the final projection matrix centered on the camera is.

$M_{proj} = M_{ortho}M_{persp}M_{camera}$


What does "6-separating" and "26-separating" voxelization mean? I was reading this paper about Voxelpipe, a voxelization library from NVIDIA and I found on section 2 Voxelization the terms 6-separating and 26-separating

I found this website that tries to explain the basic ideas on voxelization but it wasn't very much helpful understanding the terms mentioned.

Can anybody explain or point out to some other resource that can help me understand?

The terms have to do with the "thickness" of the voxelization. I'll illustrate with the help of a diagram about 2D line rasterization (from this unrelated question).

On the right is the typical line rasterization: the algorithm finds the one pixel nearest the line within each row (or column, depending on slope). This produces what we usually think of as a "1-pixel-thick" line. On the left is a conservative rasterization, which finds every pixel whose rectangle is touched by the line, and it produces a thicker line.

6-separating voxelization is like the thin line on the right, and 26-separating is like the thick line on the left, but in 3D. If you imagine the line is actually a triangle viewed on-edge, this is analogous to what the voxelization would look like.

Different types of voxelization may be better depending on what you're going to do with the voxelized data later. If you're using the voxels as a spatial hierarchy to find triangles that intersect a given region, you probably want the thick voxelization, as it's conservative. The thick voxelization might also be preferable for ray-marching, as the thin voxelization could be missed by a diagonal ray. On the other hand, the thin voxelization is a more faithful representation of the original surface, which is probably better for visibility tests, collision detection, fluid simulation, and suchlike.

The "n-separating" terminology is a bit unfortunate, but here's what it's getting at. Imagine you're doing a 3D flood-fill in the voxel grid, but in the flood-fill you only look at the 6 direct neighbors of each voxel (±1 step along each axis). Then the "6-separating" (thin) voxelization will stop the flood-fill: it suffices to separate the two sides of the surface, if only 6 neighbors are considered. On the other hand, suppose your flood-fill was allowed to go to diagonal neighbors as well—26 neighbors in all (3×3×3 neighborhood of voxels). Then the 6-separating voxelization wouldn't stop the flood fill, but the 26-separating (thick) one would.


Very efficient/fast marching cube implementation I was wondering if there's any known technique/algorithm that would compute a mesh generated by the marching cubes but hierarchically (like a divide and conquer strategy), in the implementation I know (that follows the original paper) the marching cubes is an incremental algorithm that iterates through all the voxels (regardless if they're empty or not) and it will produce from 1 to 4 triangles per voxel.

Either this or a very fast implementation of marching cubes is fine.

I cannot manage to find any reference.

This seems to be eluding to a Marching Cubes LOD algorithm such as: Place the entire volume in one giant cube. Break that volume into NxNxN cubes. And continue doing so until the cubes are at the finest granularity needed for the density function about 8 level works. Each volume then responds with either: No voxels - stop processing that sub volume OR Yes there are voxels -- process the sub volumes and keeping doing this until:  the voxel level is reached. Finally set a LOD - such as equating a screen space distance range to each subvolume level and treat the subvolume for that LOD as the final voxel. Cracks are a major issue with this algorithm and why you won't see many implementation of it.

There is also Eric Lengyel's Transvoxel algorithm: https://transvoxel.org

and here is a link an implementation of it
https://github.com/stoyannk/voxels


Modelling Young's double slit experiment Young's double slit experiment is very simple to set up and simple to explain, but it is an example of both diffraction and interference, neither of which are modelled by conventional raytracing.

It is straightforward to render an approximation of the result using textures, but that would require knowing in advance what the result should be. For an arbitrary set up, where the number and arrangement of slits is not known in advance, are there any existing algorithms for modelling the effect in order to generate the correct resulting image?

If not, what would a model need to include in order to accurately produce these effects? Can ray tracing be adapted to use rays carrying additional information, or would an entirely new approach be required?

It is in fact possible to augment a ray tracer to make it capable of simulating wave effects. The paper Rendering Wave Effects with Augmented Light Field
 describes one way to do it. In summary, they introduce a framework called Augmented Light Field that allows them to model wave effects with a ray-based representation. In this framework, rays can carry negative radiance in addition to positive radiance; intuitively, rays carrying negative radiance can "subtract" light from surfaces that it cannot reach because of interference effects, for example.

I haven't implemented it personally, and I can't speak for its implementation complexity, but I believe it is a good starting point.


Why isn't rasterization combined with raytracing more often? If I'm not mistaken, you can make a hybrid renderer which lets a rasterizer solve the visibility problem, then shade all of the projected shapes with a raytracing algorithm. It seems that this isn't commonly done however, what's the reason for that? Oversight?

Rasterization is sometimes used for primary rays. However, it limits greatly what you can do - depth of field, motion blur, participating media, refraction, only basic camera models, no adaptive or custom sampling, etc.


Why cant I use `#include` after upgrading my driver After updating my graphics driver and trying to run this example from John Chapman's blog I got this error

Vertex shader failed to compile with the following errors:
ERROR: 0:8: error(#105) #version must occur before any other program statement

The way that the version is defined is by including a file containing this code

#ifndef DEF_GLSL_
#define DEF_GLSL_

#version 420

...

So inside all other shader files one can see that the first line is as follows

#include "common/def.glsl"
...

I remember this working perfectly a few months ago. So what exactly has changed and the #include command is no longer valid?

The error message gives a clue in the wording:

#version must occur before any other program statement

Is there a reason that the previous 2 lines need to appear before the #version line? If not, you can avoid the error by moving this line to the beginning, since the actions of the lines are independent of their order in this instance.

The #version needing to be at the beginning of the file is not a new requirement. I don't know why you were able to run this code previously.


Can the uniform id be changed from run to run on the same machine? I have a shader which has several uniform variables:

uniform vec4 a;
uniform vec4 b;
uniform vec4 c;

I'm getting these handles using glGetUniformLocation() and storing them in the array. The game was suspended, then resumed and onSurfaceCreated() has been called. The question is: whether these handles will be valid or I need to re-get them using glGetUniformLocation() once again?

The uniform locations are set when the program is linked and are tied to the program.

This means that when you create a new program then the uniform locations can change.


How to never draw "Jaggies" in the first place considering resolution and scale Ok, I have a simulation I am trying to make. The entire simulation is drawn by circles (with gl_Point) and rectangles (gl_triangles with indices).

It would be great to add some anti aliasing especially as there scale is usually rather small.

My problem is I have seen some formulas that smooth step the edges of a shape, however I do not know how to properly consider scale.

Each sprite will have different scale to it, each rectangle will have a different width / height ratio, not to mention I am not yet passing in texture coords because they have been so far un-necesary. And this will run on multiple screen resolutions.

I guess I have two questions:

Is it possible to anti-alias a rectangle in a fragment shader without knowing the UV of each fragment since I am not yet passing them in? Even if it was possible how would I account for different ratios of rectangles?
What math can I use to ensure that no matter the device resolution, or shape size that the anti aliasing effects a maximum of 2 pixels (or whatever) from the edge of the shape?

Well you can render at higher resolution and sample down thhis is called FSAA and should just work without much change to render code. FSAA has the benefit of not having a conflation problem whereas smoothing does.

No, something has to emit this info.
You take the derivate (the gfx card does this for you) of the surface varying to determine how far the step would take you in your pixel. if its over the edge then you antialias.


Difference between Cropping, Scaling, Resizing & Changing Aspect Ration of an Image? I've got a new task related to image processing in a website. I am too confused in understanding different terms used with sizing of an image. What is the difference between:

Cropping an image
Resizing an image
Scaling an image
Changing Aspect-ratio of an image

Especially Resizing vs Scaling!!!

"Cropping an image" means some part of the image is removed in order for the image to fit a certain size. For example, removing the rightmost part of a rectangular image can result in a square.
"Resizing an image" usually means to stretch and squeeze it as needed so it fits a given size. Width and height are modified independently. If the ratio between them is not the same in the original size and in the final size, the image will appear distorted.
"Scaling an image" usually means resizing, but by the same amount horizontally and vertically. The ratio between the width and height is not modified, therefore the final image is not distorted.
The "aspect ratio" is the ratio between the width and the height. Changing the aspect ratio will perceptually stretch the image either horizontally or vertically.

Resizing might change the aspect ratio, and from the wording of your question I have assumed scaling meant not changing the aspect ratio.


What's the difference between Sample and tex2D I am mainly using Unity but I think that it's not related to this engine, and I am wondering what's the difference between using tex2D and Sample for sampling a texture.

They represent the same operation, but Sample is what it's called in D3D10 and newer versions of HLSL, while tex2D is what it's called in D3D9 HLSL, and NVIDIA's (defunct) Cg language. By the way, the operation is also called texture in GLSL.


Dx12 Vertex Buffer incorrect I'm playing around with directX 12 and I'm having some problems loading geometry in. I have a std::vector vertices definition and I do exactly what the dx12 template does with it.

  const UINT vertexBufferSize = sizeof(vertices);

  // Create the vertex buffer resource in the GPU's default heap and copy vertex data into it using the upload heap.
  // The upload resource must not be released until after the GPU has finished using it.
  Microsoft::WRL::ComPtr<ID3D12Resource> vertexBufferUpload;

  CD3DX12_HEAP_PROPERTIES defaultHeapProperties(D3D12_HEAP_TYPE_DEFAULT);
  CD3DX12_RESOURCE_DESC vertexBufferDesc = CD3DX12_RESOURCE_DESC::Buffer(vertexBufferSize);
  DX::ThrowIfFailed(d3dDevice->CreateCommittedResource(
     &defaultHeapProperties,
     D3D12_HEAP_FLAG_NONE,
     &vertexBufferDesc,
     D3D12_RESOURCE_STATE_COPY_DEST,
     nullptr,
     IID_PPV_ARGS(&mVertexBuffer)));

  CD3DX12_HEAP_PROPERTIES uploadHeapProperties(D3D12_HEAP_TYPE_UPLOAD);
  DX::ThrowIfFailed(d3dDevice->CreateCommittedResource(
     &uploadHeapProperties,
     D3D12_HEAP_FLAG_NONE,
     &vertexBufferDesc,
     D3D12_RESOURCE_STATE_GENERIC_READ,
     nullptr,
     IID_PPV_ARGS(&vertexBufferUpload)));

  mVertexBuffer->SetName(L"Vertex Buffer Resource");
  vertexBufferUpload->SetName(L"Vertex Buffer Upload Resource");

  // Upload the vertex buffer to the GPU.
  {
     D3D12_SUBRESOURCE_DATA vertexData = {};
     vertexData.pData = reinterpret_cast<BYTE*>(vertices.data());
     vertexData.RowPitch = vertexBufferSize;
     vertexData.SlicePitch = vertexData.RowPitch;

     UpdateSubresources(mCommandList.Get(), mVertexBuffer.Get(), vertexBufferUpload.Get(), 0, 0, 1, &vertexData);

     CD3DX12_RESOURCE_BARRIER vertexBufferResourceBarrier =
        CD3DX12_RESOURCE_BARRIER::Transition(mVertexBuffer.Get(), D3D12_RESOURCE_STATE_COPY_DEST, D3D12_RESOURCE_STATE_VERTEX_AND_CONSTANT_BUFFER);
     mCommandList->ResourceBarrier(1, &vertexBufferResourceBarrier);
  }

But this is what I get in the graphics debugger when it was suppossed to be a sphere. I can confirm that the data in vertices is correct.

If I change vertices to be what the template code had it works and renders the default cube. The only real difference mine is using a vector instead of an array.

VertexPositionColor cubeVertices[] =
{
    { XMFLOAT3(-0.5f, -0.5f, -0.5f), XMFLOAT3(0.0f, 0.0f, 0.0f) },
    { XMFLOAT3(-0.5f, -0.5f,  0.5f), XMFLOAT3(0.0f, 0.0f, 1.0f) },
    { XMFLOAT3(-0.5f,  0.5f, -0.5f), XMFLOAT3(0.0f, 1.0f, 0.0f) },
    { XMFLOAT3(-0.5f,  0.5f,  0.5f), XMFLOAT3(0.0f, 1.0f, 1.0f) },
    { XMFLOAT3(0.5f, -0.5f, -0.5f), XMFLOAT3(1.0f, 0.0f, 0.0f) },
    { XMFLOAT3(0.5f, -0.5f,  0.5f), XMFLOAT3(1.0f, 0.0f, 1.0f) },
    { XMFLOAT3(0.5f,  0.5f, -0.5f), XMFLOAT3(1.0f, 1.0f, 0.0f) },
    { XMFLOAT3(0.5f,  0.5f,  0.5f), XMFLOAT3(1.0f, 1.0f, 1.0f) },
};

Vector has a few different semantics from static arrays. For one it's a struct containing a pointer a capacity and a length (at the very least). That means that sizeof will not reflect how much data is actually stored in there (that only works on static arrays).

If you need to get the size in bytes of the data then you need

const UINT vertexBufferSize = sizeof(vertices::value_type) * vertices.size();


Type conversion to GL-types If I have a std::vector<Foo> and want to store its size in a GLuint, my compiler gives a warning that there is possible loss of data.

GLuint size = vec.size(); // "conversion from 'size_t' to 'GLuint', possible loss of data"

How do I correctly do this type of casting?
Are there more pitfalls of this kind with for example float to GLfloat?

Practically speaking, probably not - try printing the size of each type using sizeof. Probably GLuint is 32 bits and size_t is 64 bits - so 32 bits are thrown out; but float and GLfloat are probably both 32 bit IEEE 754 numbers. "Probably" sprinkled above, because technically speaking the sizes of C ints, floats and the float format are compiler dependent, but GL type sizes (and formats) should be standard (see https://www.opengl.org/wiki/OpenGL_Type). Chances are, GLfloat is just typedef'd as a float.

You may be able to turn off this warning in your compiler - I choose to explicitly cast ("GLuint size = GLuint(vec.size())" works) so I am reminded of what's happening.


Modern Screen Space Ambient Occlusion Techniques Down the road, I'm planning to implement one or another GPU based ambient occlusion technique. My goal will be to approximate a physically based offline renderer, so I was pretty happy when Wikipedia led me to Approximating Dynamic Global Illumination in Image Space, which describes SSDO. Based on the references, it looks to be from the 2008 era. Are there more modern approaches with similar accuracy goals?

This bachelor thesis briefly reviews six SSAO techniques. It could be a good start.

CryEngine 2 AO
StarCraft II AO
HBAO
Volumetric Obscurance
Alchemy AO
Unreal Engine 4 AO

http://frederikaalund.com/a-comparative-study-of-screen-space-ambient-occlusion-methods/


Why is the color in the cube being weirdly swapped? I am implementing a trackball, i.e. a camera orbiting about a fixed point, in my case the origin. When I do left click with the mouse and start moving it, I compute delta values for the angles of a sphere centered at the origin given the displacement in x and y directions of the mouse. My implementation is as follows

void SimplePerspectiveCamera::
mouseMove(const double& mouseX, const double& mouseY)
{
    if(m_setPan)
    {
        float sensitivity = 0.0075f;
        float newX = (mouseX - m_prevX) * sensitivity;
        float newY = (mouseY - m_prevY) * sensitivity;
        // TODO: Understand if it is necessary to reset the up vector to (0,1,0)
        m_up = Vector(0.0f, 1.0f, 0.0f);
        setLookAt(m_lookAt + Vector(-newX, newY, 0.0f));
        setPosition(m_position + Vector(-newX, newY, 0.0f));
        m_prevX = mouseX;
        m_prevY = mouseY;
    }
    if(m_setRotate)
    {
        float sensitivity = 0.005f;
        float newPhi = (mouseX - m_prevX) * sensitivity;
        float newTheta = (mouseY - m_prevY) * sensitivity;
        // Transform to spherical coordinates to use mouse move as deltas for
        // the angles.
        float r = m_position.norm();
        float theta = std::acos(m_position.y / r);
        float phi = std::atan(m_position.x / m_position.z);

        theta += newTheta;
        phi += newPhi;

        m_up = Vector(0.0f, 1.0f, 0.0f);
        Vector newPosition(r*std::sin(theta)*std::sin(phi), r*std::cos(theta),
                r*std::sin(theta)*std::cos(phi));
        setPosition(newPosition);
        m_prevX = mouseX;
        m_prevY = mouseY;
    }
}

What I was expecting is to be able to rotate the camera around the cube in the scene so that I can see all around it. My code almost achieves this, but I don't understand why after rotating a certain amount the camera jumps to the exact opposite side of the cube.

You can check what weird swap I am talking about this link

What am I missing so that the rotation around the object goes all the way around it?

It is likely that your specific problem arises from a singularity in atan when m_position.z goes to 0. Rather than try to debug that, I'll point out that you don't need to transform your camera position into an angle to then modify it: rather, just create a rotation from your newPhi and newTheta angles, in the form of a rotation matrix or quaternion, and then apply it to your vector, directly rotating it without doing the perilous round-trip to an angle.


Sharing code between multiple GLSL shaders I often find myself copy-pasting code between several shaders. This includes both certain computations or data shared between all shaders in a single pipeline, and common computations which all of my vertex shaders need (or any other stage).

Of course, that's horrible practice: if I need to change the code anywhere, I need to make sure I change it everywhere else.

Is there an accepted best practice for keeping DRY? Do people just prepend a single common file to all their shaders? Do they write their own rudimentary C-style preprocessor which parses #include directives? If there are accepted patterns in the industry, I'd like to follow them.

There's a bunch of a approaches, but none is perfect.

It's possible to share code by using glAttachShader to combine shaders, but this doesn't make it possible to share things like struct declarations or #define-d constants. It does work for sharing functions.

Some people like to use the array of strings passed to glShaderSource as a way to prepend common definitions before your code, but this has some disadvantages:

It's harder to control what needs to be included from within the shader (you need a separate system for this.)
It means the shader author cannot specify the GLSL #version, due to the following statement in the GLSL spec:

  The #version directive must occur in a shader before anything else, except for comments and white space.

Due to this statement, glShaderSource cannot be used to prepend text before the #version declarations. This means that the #version line needs to be included in your glShaderSource arguments, which means that your GLSL compiler interface needs to somehow be told what version of GLSL is expected to be used. Additionally, not specifying a #version will make the GLSL compiler default to using GLSL version 1.10. If you want to let shader authors specify the #version within the script in a standard way, then you need to somehow insert #include-s after the #version statement. This could be done by explicitly parsing the GLSL shader to find the #version string (if present) and make your inclusions after it, but having access to an #include directive might be preferable to control more easily when those inclusions need to be made. On the other hand, since GLSL ignores comments before the #version line, you could add metadata for includes within comments at the top of your file (yuck.)

The question now is: Is there a standard solution for #include, or do you need to roll your own preprocessor extension?

There is the GL_ARB_shading_language_include extension, but it has some drawbacks:

It is only supported by NVIDIA (http://delphigl.de/glcapsviewer/listreports2.php?listreportsbyextension=GL_ARB_shading_language_include)
It works by specifying the include strings ahead of time. Therefore, before compiling, you need to specify that the string "/buffers.glsl" (as used in #include "/buffers.glsl") corresponds to the contents of the file buffer.glsl (which you have loaded previously).
As you may have noticed in point (2), your paths need to start with "/", like Linux-style absolute paths. This notation is generally unfamiliar to C programmers, and means you can't specify relative paths.

A common design is to implement your own #include mechanism, but this can be tricky since you also need to parse (and evaluate) other preprocessor instructions like #if in order to properly handle conditional compilation (like header guards.)

If you implement your own #include, you also have some liberties in how you want to implement it:

You could pass strings ahead of time (like GL_ARB_shading_language_include).
You could specify an include callback (this is done by DirectX's D3DCompiler library.)
You could implement a system that always reads directly from the filesystem, as done in typical C applications.

As a simplification, you can automatically insert header guards for each include in your preprocessing layer, so your processor layer looks like:

if (#include and not_included_yet) include_file();

(Credit to Trent Reed for showing me the above technique.)

In conclusion, there exists no automatic, standard, and simple solution. In a future solution, you could use some SPIR-V OpenGL interface, in which case the GLSL to SPIR-V compiler could be outside of the GL API. Having the compiler outside the OpenGL runtime greatly simplifies implementing things like #include since it's a more appropriate place to interface with the filesystem. I believe the current widespread method is to just implement a custom preprocessor that works in a way any C programmer should be familiar with.


What is the state-of-the-art on using computers to "clean-up" images? TWO PART QUESTION - We've all seen the movies/TV shows where the police/feds/spies use computer software to take a grainy photo and do a "clean-up" to see a better picture and more details. I assume the concept is executed by some sort of uber-power pixel-smoothing or anti-aliasing type of algorithms to fill in the blanks based on deductive processing.

PART 1: How real is this technology in the public/commercial software world? I'm not asking about any speculation on alleged secret gov software or such, I just want to know where we actually are with this concept today? How much is fully automated vs human-assisted.

PART 2: Assuming there actually is some reality with this technology for photographs the second part of this question is how (if at all) has this been applied to videos? Again the issue of fully automated vs human assisted is of interest here.

At the heart of this post is the ultimate question of how viable is today's software for being able to take an old VHS or DVD recording and process the frames to create a new HD-resolution remaster. Considering that doing this would mean cleaning up tens-of-thousands of frames for even a simple wedding video I am not expecting this technology to be fast of course.

NOTE: Per in-topic and meta discussions I went ahead and cross-posted this in two other suggested SE forums to acquire their viewpoints and expertise on this matter. So far (it is still a little "early in the day" so to speak) I have received some pretty interesting information in 2 of the 3.

When this is all done (when good answers have been selected) I would appreciate a way to merge these for the benefit of all three SE communities:

Computer Graphics: What is the state-of-the-art on using computers to "clean-up" images?
Signal Processing: https://dsp.stackexchange.com/questions/28168
Video Production: https://video.stackexchange.com/questions/17363

The closest thing I know of to the "computer: enhance!" trope in real life is the family of Single Image Super-Resolution techniques. That page shows a number of examples of the results on various images. You can see that while it improves the visual quality of the enlarged images, it's a long way from what you see on TV where they can read the text on a letter that's reflected in a wine glass, or recognize the face of someone standing in the shadows, etc.

The technique basically works by observing that a single image often contains recurring instances of the same patterns or structures. When you have multiple copies of the same pattern, they often occur at different sub-pixel offsets, i.e. aligned differently to the pixel grid, which means that each copy contains slightly different information about the underlying pattern, and by putting them together you can recover (really, guess at) a higher-resolution version of that pattern. This can then be used to "fill in the missing detail" wherever that pattern occurs. This is a very handwavy explanation, but you can see the paper (linked from the above site) for more details. Most of it is automated, but I'd guess it still requires a fair amount of human parameter-tuning to get good results.

The paper is from 2009, and it looks like there have been a few follow-ups since then, but only incremental improvements.

Also note that if you have a specific use case such as reading vehicle license plates, I'd guess it's possible to use machine-learning techniques to get much better results than you can in general. In that case, you have a concrete, known set of possible shapes (the characters that can appear on a license plate, in the particular font used for license plates), and rather than "enhancing" an image of literally anything, you're just trying to find the set of characters that best matches the image you're looking at. That's not my area of expertise, though.


What happened to my texture mapping? Recently, I am writing a simple software rasterizer for self-studying and fun. But the texture mapping has problems. The texture I want to map is below:

But the final image of two triangles is problematic, it shows below:

The triangle rasterizing algorithm I use is the Triangle barycentric coordinates interpolation. It seems the mapping of second triangle is not correct. My code is below, As the code shows, the three vertices of the triangle is, p1, p2, p3.

I hope someone can help me to fix the problem. Thanks very much!

if (mRenderState & (RENDERING_TEXTURE | RENDERING_COLOR))
{
    int x0 = p1.x();
    int x1 = p2.x();
    int x2 = p3.x();
    int y0 = p1.y();
    int y1 = p2.y();
    int y2 = p3.y();

    //Firstly build the bounding box
    int xMin = std::min(x0, std::min(x1, x2));
    int xMax = std::max(x0, std::max(x1, x2));
    int yMin = std::min(y0, std::min(y1, y2));
    int yMax = std::max(y0, std::max(y1, y2));

    // Triangle setup.
    // Compute the edge equation coefficients.
    float edge_a_0 = -(y1 - y0);
    float edge_a_1 = -(y2 - y1);
    float edge_a_2 = -(y0 - y2);

    float edge_b_0 =   x1 - x0;
    float edge_b_1 =   x2 - x1;
    float edge_b_2 =   x0 - x2;

    float edge_c_0 = edge_a_0 * -x0 + edge_b_0 * -y0;
    float edge_c_1 = edge_a_1 * -x1 + edge_b_1 * -y1;
    float edge_c_2 = edge_a_2 * -x2 + edge_b_2 * -y2;

    // Triangle area and inverse area
    float area = 0.5f * ((x1 - x0) * (y2 - y0) - (y1 - y0) * (x2 - x0));
    float inv2Area = 1.f / (2.f * area);

    // Check the triangle is backfacing.
    if (area <= 0.)
        return;

    // Loop over the bounding box of the pixels that the triangle possibly
    // covers.
    int py;
    int px;
    for (int py = yMin; py < yMax; ++py)
    for (int px = xMin; px < xMax; ++px)
    {
        float e0 = edge_a_0 * px + edge_b_0 * py + edge_c_0;
        float e1 = edge_a_1 * px + edge_b_1 * py + edge_c_1;
        float e2 = edge_a_2 * px + edge_b_2 * py + edge_c_2;

        //Check the sample is outside the triangle
        if (e0 <= 0.f || e1 <= 0.f || e2 <= 0.f)
            continue;

        float wt0 = e1 * inv2Area;
        float wt1 = e2 * inv2Area;
        float wt2 = e0 * inv2Area;

        // Interpolate z
        float z = (wt0 * p1.z() + wt1 * p2.z() + wt2 * p3.z());
        float zrhw = 1.0f / z;

        if (z < mZBuffer[py*mWidth + px])
        {
            mZBuffer[py*mWidth + px] = z;

            if (mRenderState & RENDERING_COLOR)
            {
                float r = wt0 * v1->r + wt1 * v2->r + wt2 * v3->r;
                float g = wt0 * v1->g + wt1 * v2->g + wt2 * v3->g;
                float b = wt0 * v1->b + wt1 * v2->b + wt2 * v3->b;
                DrawPixel(px, py, Color(r, g, b));
            }

            if (mRenderState & RENDERING_TEXTURE)
            {
                float u = wt0 * v1->u + wt1 * v2->u + wt2 * v3->u;
                float v = wt0 * v1->v + wt1 * v2->v + wt2 * v3->v;
                unsigned int cc = TextureRead(u, v);
                unsigned char r = (cc & (0xff << 16)) >> 16;
                unsigned char g = (cc & (0xff << 8 )) >> 8;
                unsigned char b = cc & (0xff);
                Color color;
                color.setColor(r, g, b);
                DrawPixel(px, py, color);
            }
        }
    }
}

You aren't doing perspective correct texturing. At each pixel, $(X,Y)$, you need to be computing U and V with the equivalent of:

$U=\frac{aX+bY+c}{pX+qY+r}$

$V=\frac{dX+eY+f}{pX+qY+r}$

The denominator of both is linearly interpolating 1/W, while the numerators are, respectively, interpolating U/W and V/W.

I.E. for each triangle vertex you need to compute 1/W, U/W and V/W and then interpolate these. You can do it via barycentrics if you wish but you must do the division per pixel.


Multiple swapchains in Vulkan app with ImGui I am integrating imgui into my app using glfw + vulkan example.

The function vkCreateSwapchainKHR() is also called by imgui setup logic in addition to my app on the same device and it fails with "internal drawable creation failed".

Under what circumstances, if any, would this be a valid approach?

Should my app and imgui share the same swapchain?

The Vulkan specification is very clear on this:

  A native window cannot be associated with more than one non-retired swapchain at a time.

You can create a new swapchain for a surface, but only by passing the old swapchain as the oldSwapchain parameter, which will cause the old swapchain to enter the retired state.

So yes, both you and imGUI should get on the same page about who owns the swapchain.


Point light illuminating only half of a scene A point light is illuminating only a half of a scene in my small 3D renderer. It happens on every scene I've tried (conference room, cornell box, etc).

How do I go about debugging this? What could be wrong?

Here are the shaders. I'm not doing anything out of the ordinary. Light is positioned at (0, 1.0, 0) in world space.

//
// VERT SHADER
//
#version 410 core

uniform mat4 M;
uniform mat4 V;
uniform mat4 P;

layout (location = 0) in vec3 position;
layout (location = 1) in vec3 normal;
layout (location = 2) in vec3 color;
layout (location = 3) in vec2 texCoords;

out vec3 fWorldPos;
out vec3 fNormal;
out vec3 fColor;
out vec2 fTexCoords;

void main()
{
    mat4 normalMatrix = transpose(inverse(M));

    fWorldPos = vec3(M * vec4(position, 1.0));
    fNormal = normal; // also tested multiplying with normal, M, and MV matrices
    fColor = color;
    fTexCoords = texCoords;

    gl_Position = P * V * M * vec4(position, 1);
}

//
// FRAG SHADER
//
#version 410 core
#define MAX_POINT_LIGHTS 4

struct PointLight {
    float strength;
    vec3 position;
    vec3 color;
    vec3 attenuation;
};

struct Material {
    vec3 color;
};

uniform Material material;
uniform int totalPointLights;
uniform PointLight pointLights[MAX_POINT_LIGHTS];
uniform vec3 cameraPosition;

in vec3 fWorldPos;
in vec3 fNormal;
in vec3 fColor;
in vec2 fTexCoords;

out vec4 color;

float attenuate(float dist, const PointLight light) {
    return light.strength / (light.attenuation.x + light.attenuation.y * dist + light.attenuation.z * dist * dist);
}

//
// MAIN
//
void main()
{
  vec3 normal = normalize(fNormal);

  PointLight light = pointLights[0];

  vec3 lightDir = light.position - fWorldPos;
  float lightDst = length(lightDir);
  lightDir = lightDir / lightDst;

  float att = attenuate(lightDst, light);
  float diffuseFactor = max(dot(normal, lightDir), 0.0);

  color = vec4(material.color * diffuseFactor * att, 1);
}
```

The normals were indeed wrong. Which was crazy, because I had even checked in Blender that they were correct (and they were). Turns out, when loading the mesh, I was passing the normal as color and vice versa. Valletin's normal visualization shader was great (https://vallentin.dev/2015/02/16/visualizing-normals).

Thanks for your help!


Uses for Stencil Buffer I've been doing OpenGL tutorials using LearnOpenGL. Currently I'm reading about Stencil Buffers and in his example he uses a Stencil Buffer to draw an outline around an object. I've also read that it can be used to create shadows and reflections. My question is, what are other practical uses for a Stencil Buffer and why were they invented in the first place?

The original use for the stencil buffer was for non-3D content. Say you're making a 3D game. Most of the screen is the 3D content, but you also have some GUI around the outside which is not drawn through OpenGL but through a 2D canvas API. You can mask off the shape of the 2D GUI using the stencil buffer, and then make sure that your 3D/OpenGL content won't affect it and won't cause overdraw (which wastes fragment shading time).

As you've seen, it's also used for techniques like shadow volumes and some reflection techniques. In general, it can be used for techniques where you draw an object and then want to run another fragment shader (or some more draw calls) to alter that object's appearance, by adding reflections, lighting, or screen-space effects. In that case, when you draw the object the first time, it's very cheap to fill the stencil buffer at the same time. Then, your later pass (to add the reflections or whatever) can use the stencil test to only run on those fragments that eventually did get drawn on the screen.

Another common use is when rendering a scene in multiple parts. For example, in a flight simulator, it's common for the cockpit (and aircraft exterior) and the terrain (and other aircraft) to be done in completely separate passes. This lets you use a lower-precision depth buffer without causing z fighting, because you can use different depth ranges for the near content and the far content. But if you do this, you can't reuse the same depth values for both passes. In this case, you can fill the stencil buffer when you're drawing the cockpit, and use the stencil test to avoid overdrawing the cockpit when you draw the terrain. (You need to do more to handle glass effects on the cockpit windows: this description is an oversimplification.) You can use similar techniques for drawing the player's weapon and/or hands in first-person shooters.

The stencil buffer is used in many techniques: too many to list them all individually. But in general, if you want to have some render passes only affect some fragments, and the fragments you want to affect are those corresponding to particular objects or shaders, then the stencil buffer is a cheap way to enable that.


What is the difference between a Sampler and an Image variables in GLSL? Reading through some code about voxelization I found the following line in the fragment shader

layout (binding = 0, r32f ) coherent uniform writeonly image3D volumeTexture;

I have only used samplers in the past and have never seen image3D before.

I have read the entries about Sampler Varibles and Image Variables but I really don't grasp the differences, advantages/disadvantages between Sampler and Image.

Can someone help me pointing out what are the difference between these two types of variables and when should I used one over the other?

Image and Samplers differ in many ways. Unlike Samplers, images can only be addressed with an integer coordinate that ranges from  (0,0) to (x,y), where x and y are the width and height of the image. They support random access reads like texelFetch, and support no filtering as you're accessing raw texels. The most interesting thing with images is that they support read and write access, in Open GLES there are limitations, wherein an image may only be specified as readonly or writeonly, not both, in desktop GL though you may specify an image as read/write so that you can both read and write to that image in a single pass.

Using a sampler, if you wanted to write a color into it you'd have to use an FBO, then a second pass to consume that. With images, you can do this all in one pass by writing to the image and reading from it, you'd have to handle the synchronisation yourself of course.
Images are very useful for image processing etc.


Raytracing in OpenGL using Compute Shaders I Recently posted this question on SO but didn't got any response so i thought to post it here since it's somewhat related to Raytracing.

I am making a real time ray tracer in OpenGL using Compute Shaders for my project and was following this link as a reference. The link tells to first draw a full screen quad, then store all the individual pixel colors gotten through intersections in a texture and render the texture to the quad.

However i was thinking can't we use Frame Buffer Objects to display the texture image instead of rendering the quad and save the over head?

Like I save all the colors using ImageStore and GlBindImageTexture in a texture, then attach it to a FBO to display it. And since I won't be using any rendering commands I won't be causing a Feedback loop as in writing and reading the same texture?

Here is the snippet

  void initialization() {
  int tex = create new texture object

  setup min/mag NEAR filtering for that texture
  initialize texture as 2D rgba32f FLOAT and some width/height

  int vao = create VertexArrayObject of full-screen-quad
  // This quad is used to render our
  // "framebuffer" texture onto the screen.

  int computeProgram = create-and-link program with single compute shader object

  int quadProgram = create-and-link simple full-screen quad vertex and fragment shader
  // The fragment shader would fetch
  // texels from our "tex" texture.

  Setup constant uniforms in quad program, such as "tex" texture unit = 0
}

 The rendering process to produce one frame will be as follows:

Bind the Compute Shader
Setup camera properties in compute shader uniforms
Bind level 0 of framebuffer texture to image binding point 0
Start/dispatch the compute shader to generate a frame in the framebuffer image
Unbind image binding point
Bind the written framebuffer texture in OpenGL to texture unit 0
Bind the full-screen shader program
Bind the full-screen-quad VAO
Draw the VAO
Release all bindings

So i searched a lot after this and I think it was my confusion on FrameBuffer Objects.

I thought you could use FBO's just like a default FrameBuffer and display the texture image attached to it but you can't. It's only used for offscreen rendering. So while you can use rendering commands to draw something to a "texture image" attached to it, you can't "display the image" by making it default framebuffer or something like that.


What prevents older GPUs from supporting Vulkan? More out of curiosity than for any practical purpose, I'm wondering what is lacking from older GPUs that causes them to be unable to support Vulkan.

I know that some argue that some hardware, such as nVidia's Fermi series of GPUs, could support Vulkan in hardware but the vendors simply won't implement it for business reasons, but even disregarding that, there is obviously some most recent hardware that really doesn't support Vulkan, and I'm curious what they lack. Like, say, the nVidia Tesla GPUs, AMD's TeraScale (VLIW) GPUs, or Intel's Gen8 hardware.

I'm mostly curious in knowing if the features in Vulkan that precludes support for these GPUs are somewhat "peripheral" in the API (like, say, required support for some particular texture format that the API could have just as well not required but did anyway to create a baseline that was considered reasonable), or if it's truly central to the Vulkan model.

So, I'll start by saying I'm not a driver developer, but I have read many comments and docs in this regard.
First of all, we can see how Khronos's own slides on presentation day were mentioning some rather vague "any OpenGL ES 3.1 GPU" (or desktop 4.X). This means something added around that mark would be the secret to it. Compute shaders are definitively [at least one of] the things, being confirmed here, by an Intel developer, and in the spec with no possible unsupported value for MaxComputeWorkGroupSize and this likely smoking gun:

If an implementation exposes any queue family that supports graphics
operations, at least one queue family of at least one physical device
exposed by the implementation must support both graphics and
compute operations.

In addition to that though, it could also be there are other mandatory capabilities (such as indirect draws or load&store, mentioned in the comments above too). But I have some slight difficulty interpreting the document. With the exception of robustBufferAccess, it seems like just about anything else big you could imagine is optional (from anisotropic filter, to geometry and tessellation shaders, to any special number format other than good old float32)
This being the most hypothetically simple case then, doesn't mean it's all fine and dandy. Not having virtual memory will make your life a hell for example, and lack of other architectural features may basically nullify whatever theoretical advantage in speed the API has.
These links should also answer your wonders about ATi cards. Likely same story for NVidia then (aside of perhaps some microbenchmarks, I'm not sure how much really workable Fermi's 2-years-delayed bare-bones DX12 support eventually is). As for Intel, I don't know what you are talking about, Gen8 is supported since Day1 and in fact even Gen7 shakily works now.
Going even further back, I guess if any the question becomes "what's so special about compute shaders again".
And, well, the thing isn't probably the very specific feature itself (because it's not that you couldn't already do some sort of general purpose calculation even before the unified shader model), but more like the whole general GPU architecture needed to support it. Check this for some backstory.
Before DX11 raised the bar for example, just think that you couldn't even run more than a single "task" on a Tesla GPU, without having to call a context switch (with comparable limitations also likely existing for TeraScale 1 and Gen6). And may God save us from when pixel and vertex units were separated hardware.
fun fact: it turns out that if you are a low-end kind-of-modern mobile gpu (rather than an old desktop card from a time when certain concepts hadn't even been invented yet), engineers can cut you down so much that even if you just sport D3D 9_3 feature level that's still potentially enough to support Vulkan.
EDIT: for such somewhat twisted reasons, it can also happen that while you are just GLES 2.0 compliant (and compute shaders themselves aren't supported atm, even though the architecture would be capable) a restricted version of Vulkan could also be possible. It doesn't give you everything, but it can still lead to benefits over just "old apis".


Average point on sphere between 3 other points (spherical triangle) I am trying to get a normal of a triangle on a sphere, however the normal equations (straight line) give the wrong answer.

For the points:

$$(-986.786743, -93.348259, -132.439041)\\
(210.731644, -534.785583, -818.290039)\\
(637.734985, -449.826904, -553.501648)$$

on a radius $1000$ sphere, the average point (straight line) is $(-0.074536, -0.580873, -0.810574) * 1000$, which is visibly incorrect on a sphere.

The triangle's normal normalized to distance $1000$ gives: $(0.086455, 0.899604, -0.428062) * 1000$ which is different from the average value above (which would have also given a normal) and also far off from the spherical triangle.

Edit: I was trying to unwrap a mesh into a globe to map it to a texture. I needed to know if triangles were upside down using normals.

If i read you right, you have a triangular patch which is on the surface of a sphere - so the triangle isn't really a true triangle in that it isn't flat, but instead is just made up of 3 points on a sphere connected by lines across the spheres surface.

Also if I understand you correctly, you have a point on this triangle that you want to get the normal at.

If all that is right, the way to get the normal is to subject subtract the origin from your point on the triangle (since it is also a point on the sphere) and normalize it.

The normal at any point on a sphere is parallel to the point from the center of the sphere to that point on the surface.


Tasks of a professional graphics programmer? Currently I'm 16 and in high school and I love graphics programming and I'm seriously considering it as a career path. I was wondering what a day as a graphics programmer looks like for offline Rendering (path tracing).

Since were close to hitting photorealism now it seems like the only main challenge for the next decade is performance. I've been digging through the recent SIGGRAPH papers for the Future directions for rendering but haven't found much.

So what kinda tasks can graphics programmers do these days? Working on performance can't be the only thing.

Also, is a lot of the work research based as opposed to traditional programming?

I don't know if this is exactly what you're looking for. I work tangentially in the film and TV industry. I don't work for a studio, but I work on the software that studios use for their productions. It's software that any user could buy if they wanted to, but it has been used in feature films, television shows, commercials, music videos, and more.

About half of my time is spent programming and debugging. This is the fun stuff. It involves implementing rendering and filters of various types, sometimes based on papers I've read from SIGGRAPH or cool web sites or book. I don't do very much user interface work, though there is some. I work with OpenGL and Metal mostly. I need to know image processing algorithms in addition to general computer science algorithms, like graph processing and traversing, for example. This includes dealing with source control, continuous integration, stepping through code to find and fix bugs, talking to QA about problems, etc.

The other half of my time is spent doing non-programming things. This is a variety of things like attending conferences to learn (things like the Game Developers Conference or SIGGRAPH), conventions to promote the company and/or product (things like NAB, IBC, etc.), answering tech support emails and writing sample code (we have an API so 3rd parties can extend our product), attending meetings (scrum, management updates, company-wide meetings, etc.), talking with marketing about how our new features work, meeting with customers to understand how they use the product, explaining to our documentation people how the product works so they can document it, etc.

This seems like a pretty similar breakdown to other programming jobs in other industries from talking to my friends who work in other areas.


Brightness and contrast VS Gamma, is it possible to identify the gamma "correction amount"? Trichoplax question aroused my curiosity and the answers in this question also reminded me why I sometimes use different gamma "amounts" to enhance images.

Therefore, is it possible to determine the amount of gamma applied to an image by knowing its source (the original image)? I.e. can the mathematical formula be applied to compare two images and determine the difference in gamma "amount"?

Example: which of the following have a different gamma amount (I'll give you a non-edible cookie if you find out, and sorry trichoplax for snatching your ball):

If I get correctly what you are asking you basically just need to find the G in this equation:

$$Image_{out} = Image_{in}^G$$

This could be easily solved as

$$G = \frac{\log{Image_{out}}}{\log{Image_{in}}}$$

Because usually gamma is applied in a uniform fashion on the image, you can just pick any two non zero pixel values (one for source and one for destination) to find out the gamma value applied.

EDIT: As @ChristianRau pointed out, if you don't actually know if the image has been modified with gamma correction, you have to take an higher amount of samples and trying to fit a gamma function on those. If a fit to a gamma function results in too many significant outliers, then probably gamma correction wasn't the function applied.


Phong BRDF in Mitsuba, Tungsten and AGI In the book AGI (Advanced Global Illumination), the Phong BRDF is

$f_r(x,\Psi\leftrightarrow\Theta)=k_s\cfrac{(R\cdot\Theta)^n}{N\cdot\Psi}+k_d$

$k_d=\cfrac{albedo}\pi$

In Mitsuba and Tungsten, the Phong BRDF is

$f_r(x,\Psi\leftrightarrow\Theta)=[\cfrac{ratio_{diffuse}}\pi+\cfrac{(R\cdot\Theta)^n(n+2)(1-ratio_{diffuse})}{2\pi}]\cdot albedo$

Why there is a $\cfrac{n+2}{2\pi}$ in the specular part?

What's the relation between the two BRDFs?

It's the normalization factor to make sure the BRDF always reflects the same amount of energy regardless of the value of the specular exponent n. Without that factor, changing the specular exponent changes the overall reflectance of the material.

Further reading: The Blinn-Phong Normalization Zoo | The Tenth Planet Blog


Improved Laplacian smoothing of noisy surface meshes I have a question regarding Laplacian Smoothing of meshes and in particular the paper "Improved Laplacian smoothing of noisy surface meshes" by J. Vollmer in the early 2000's.

What is the reason for shrinkage after Laplacian Smoothing on noisy surface meshes?

This is a quite common property of smoothing in 3D. When you have some data and you smooth it out you get some kind of average of the local variation. And this works fine, and stably, in one dimensional graphs.

But when your entire data set is being smoothed over its coordinates the neighboring coordinates drag your data with them. Those again are dragged by their neighbors. The net effect is that the local variation of the surface as a whole is dragging itself towards an average that is stable in the smoothing operation itself. This will cause shrinkage because a closed object as a whole has a local average that is towards the common center of all the points in operation. Even in cases where you go locally outwards you are eventually caught up with the edges that drag you inwards.

In other words there is no structure to keep the surface locked to its place. The tendency to drag towards the center eventually wins because there is nothing to counteract this. There just simply are more opportunities to go inward than outward.


How to get the length of paths in an given font/character I wan't create some LED-rope text, using some script font, like the following:

For this I need somewhat estimate the needed length of the LED-rope for the given letter (or combinations of letters).

Some more info:

The font will be TTF or FT2.
I could manage some programs in perl using the FreeType library, eg. read some properties of the character in the font-file - if this helps.
Also could install Inkscape (Illustrator  open-source alternative) or even could use Illustrator at my friend - if any of this tools could help.

Looking for some suggestions how to measure the length of paths in the given font for the particular character (or whole text).

I don't need exact precision, (see the purpose above) and looking for the solution as simple as possible, but I haven't any idea how even approach this problem. :(

Of course, I always could print the text and buy some inexpensive curve runner, but would be better to have some computer-based solution. :)

Ps: I'm not fully sure that this is the right place to ask such question, if not - I will delete it, but please suggest some better place.

The way I would approach this is to get the outline curves of the font and then break each one down into some fixed number of steps (like, say, 100), and make a series of very small straight lines.

So if you've got some bezier curve, you loop t from 0 to 1 in 100 steps and plug it into the equation:

double x = calculateBezier(0.0, c0.x, c1.x, c2.x, c3.x);
double y = calculateBezier(0.0, c0.y, c1.y, c2.y, c3.y);
double dist = 0.0;
for (double t = 0.01; t <= 1.0; t+= 0.01)
{
    double nextX = calculateBezier(t, c0.x, c1.x, c2.x, c3.x);
    double nextY = calculateBezier(t, c0.y, c1.y, c2.y, c3.y);
    double nextDist = sqrt((nextX - x) * (nextX - x) + (nextY - y) * (nextY - y));
    dist += nextDist;
}

Where calculateBezier() is a function that calculates a point on a bezier curve given the parameter t and control points c0 through c3:

double calculateBezier(double t, double c0, double c1, double c2, double c3)
{
    double t1 = 1.0 - t;

    return (t1 * t1 * t1) * c0 + 3.0 * (t1 * t1) * c1 +
            3.0 * t1 * (t * t) * c2 + (t * t * t) * c3;
}

And just add the distances of each part of the curve until you're out of curves.

Note that this assumes cubic bezier curves. For quadratic, the calculateBezier function would calculate a quadratic one (like described here).


How to use Monotone cubic interpolation in 3D? I wanted to use Monotone cubic interpolation, but the site only provide explanation for 2D case. How can I extend it to 3D?

Ok, monotonic interpolation depends on what you are monotonic about. For a simple 1D function interpolation monotonicity is easy to define. But for a 2D and 3D dataset its not so self evident what the situation would be.

First you could interpolate along a independent variable t in which case your monotonicity is most probably in relation to t. This is the same as interpolating each direction separately.
Second you could interpolate along one of the axes, so that your interpolant becomes a function of position on that axis. Reducing a 2D case into a 1D case and a 3D case to two separate 2D cases.
Third you could interpolate on some other spatial variable.

So you see the question is a bit open ended and is hard to say for sure.


Visualize the interpolated unit quaternion on the surface of the unit sphere My major is mechanical, please forgive me for asking questions that may seem trivial to you.

First, I'm reading the following paper:

  Dam, Erik B., Martin Koch, and Martin Lillholm. Quaternions, interpolation and animation. Vol. 2. Copenhagen: Datalogisk Institut, Københavns Universitet, 1998.

On page 35, it says:

  Since quaternion space is four-dimensional, we cannot visualise the interpolated curves directly. We will always interpolate between unit quaternions, and the interpolated quaternions will always (with a few exceptions in chapter 6 on page 38 and 69) be unit quaternions. This means that we only need three dimensions to visualize the interpolation curves, because they lie on the surface of the unit sphere.

I don't quite get the last sentence: why only three dimensions are needed to visualize the unit quaternion or why unit quaternion lies on the surface of the unit sphere? I thought they lie on the unit hyper-sphere. Does the author assume that we are viewing the quaternions from the south pole of the hyper-sphere?

Second, I was also following this paper:

  Ramamoorthi, R., & Barr, A. H. (1997). Fast construction of accurate quaternion splines.

I found they are also treating (plotting) a unit quaternion as a point on a unit sphere $\mathbb{S}^{2}$. I wonder if you could recommend some resources in why we can view a unit quaternion $q = (a+b\mathbf{i}+c\mathbf{j}+d\mathbf{k})$ as a point on a unit sphere $\mathbb{S}^{2}$?

Any help is appreciated. Thank you.

Because given 3 values of a unit quaterion we can derive the fourth value.

Which means there exists a bijective mapping from unit quaternion to 3D space.

Though for visualization you want the mapping that makes what is happening the most easily discernible. However a usable visualization doesn't need to be fully bijective.


Algorithms to "anti-alias" (or somehow improve) binary 1-bit drawings and fonts These days anti-aliasing uses gray scale pixel values on displays with high pixel counts.

I'd like to take a step back in time and learn what is available for anti-aliasting or at least improving the quality of curved lines and fonts drawn on bindary (on/off) 1-bit displays, such as these low cost OLED displays shown below.

I don't know if the procedure would be called anti-aliasing, or something else. Ideally an explanation or a link/reference to a mathematical procedure or algorithm would be most helpful, or if something exists in Python, that would be great as well.

PIL, the Python Image Library is wonderful but I have a hunch it is mostly useful for gray scale or RGB continuous tone images rather than binary 1-bit images.

See for example the question Drawing multilingual text using PIL and saving as 1-bit and 8-bit bitmaps, where PIL turned this gray scale image:

into this 1-bit binary image:

and similar things would happen if I drew circles in PIL and then converted to 1-bit. The antialiasing is designed for continuous tone display, and so the subsequent thresholding just makes a mess.

Example of a 1-bit binary OLED display (128x64 pixels) cropped from this image from the AdaFruit Page Monochrome 0.96" 128x64 OLED graphic display, Product ID: 326.

PIL's Image.convert function performs dithering by default when you convert the image to 1-bit—not simply thresholding. That's what creates the noise along the edges of the shapes; the antialiasing isn't the problem. Unfortunately, at such a low resolution, the dithering does more harm than good.

I grabbed your original image from your other question and did a straight threshold at 128, and it comes out looking a lot better:

That's probably about as good as you'll be able to do on a low-resolution 1-bit display, absent a bitmap font that's specifically designed for those circumstances.

In PIL this operation can be implemented using Image.point, as follows:

image1Bit = image8Bit.point(lambda x: 0 if x < 128 else 1, mode='1')


Geometric interpretation of this bilinear interpolation equation? Bilinear interpolation can be achieved by interpolating two values across the x axis, and then interpolating between the results across the y axis, like in the image below (from https://blog.demofox.org/2015/04/30/bilinear-filtering-bilinear-interpolation/)

A way to write this mathematically is like this:

$x,y,P_{xy} \in \mathbb{R}$

$x,y \in [0,1]$

$z = (P_{00}(1-x) + P_{10}x)(1-y) + (P_{01}(1-x)+P_{11}x)y$

Doing some algebra, you can get it into a polynomial form:

$z = (P_{00}-P_{10}-P_{01}+P_{11})xy + (P_{10}-P_{00})x + (P_{01}-P_{00})y + P_{00}$

I'm trying to find the (geometric?) intuition for this form of bilinear interpolation but the first term doesn't make much sense to me.

The second term makes some sense because it's saying "add in the difference across the x axis, multiplied by how far you are down the x axis".  The third term is the same, but for the y axis.

The last term is just adding the value at the "starting coordinate" that everything else adds to, so that makes sense as well.

I can't figure out how that first term fits in though, or what exactly it's calculating.  it is the only term that deals with the $P_{11}$ so it is obviously doesn't something meaningful, but I can't quite figure out what it is.

Can anyone explain bilinear interpolation from this point of view?

Note: $P_{00}$ is (0,0), $P_{01}$ is (0,1), $P_{10}$ is (1,0) and $P_{11}$ is (1,1).

If we look at the two terms with only x or only y they fully describe what happens when you go right or up from the starting point.

The xy-terms does not come into play at all.

When you want to go "into the square" or along the upper or the right edge then the xy-term comes into play. It cancels out the x/y-terms and introduce the P11-value that it should obtain in upper-right.

No full explanation here, just nudging along towards insight!


Why MOCAP data don't use simple point coordinates? This might sound stupid for you but I really don't know how it works in the domain of 3d animation and motion capture, but I need to work with ASF/AMC files at the moment and I find it a little bit complex to understand (being a non-native english speaker does not help).

So the question is, why motion captures main file types don't use simple joint coordinates for the skeleton?

Edit: I am particularly thinking about Acclaim ASF/AMC format mainly because I need to use it with an imlementation of an algorithm that use joint vertex positions and deltas from initial position for the motion part. Having looked at how the data are represented it makes more sense to me to represent skeletons as joint vertex and motion as deltas between the intial skeleton and the new moved one.

Joint coordinates alone don't give you enough information about rotation.

For example, hold out your arm to your side. Notice that you still have one degree of freedom to twist the arm without affecting the positions of your elbow or wrist. Thus, if you had a motion capture format that stored only those coordinates, you would not be able to distinguish these poses.


Does Vulkan have the equivalent of OpenGL's separable shader program and pipeline? If you have a shader pipeline in OpenGL that goes something like the following:
// Pseudocode

// FragmentShader
// in vec3 inVertexPosition;
// in vec2 inTexCoords
//
// int main()
{
    // Sample texture coordinate and write to fragment out
}

Then you have another fragment shader that does something a little different:
// int main()
{
      // Sample texture, calculate noise and mix with texture colour
}

In OpenGL if you want to execute this second shader you only compile this second shader and bind to the fragment shader stage of the shader pipeline. You can then create another a different vertex shader and attach that to the vertex shader stage whenever you need it.
Let's just say I have 5 vertex shaders and 5 fragment shaders, in OpenGL (because it supports separable shader objects) you only need to compile 5 x 2 = 10 shaders. However, as in Vulkan, as far as I know it doesn't support it, which means you have to compile 25 programs if you want to use each combination.
Does Vulkan have the equivalent of the OpenGL separable shader objects, or are there any plans for it? They seem just so much more flexible.
I've thought about being able to bind shader stages in my code, and then every combination of shader stages could be mapped to a single compiled program, however I don't think the mapping could be searched efficiently. I'm not sure. It seems being forced to compile an entirely new pipeline when inserting a shader stage is wasteful when you make many shaders.
It seems in principle this is like compiling a new program every time you created a new function.

Does Vulkan have the equivalent of the OpenGL separable shader objects, or are there any plans for it? They seem just so much more flexible.

It would be "so much more flexible". But Vulkan is a low-level rendering API. It's job is to give you access to the hardware as close to the metal as possible; user-convenience isn't even in the top-5 goals for the API.
Putting all of the context state into one gigantic, immutable (more or less) object gives the implementation all the power it needs to make your code fast. Forcing implementations to deal with stitching together shaders would make implementations more complicated (and another goal of Vulkan is to be relatively easy to implement).
The best way to solve this (where possible) is to just have fewer shader variations. Use uniforms/push-constants to detect what variations you want to use, then have conditionals based on them. Obviously this can't be done in every situation, but the uber-shader approach is the way to handle it to the degree possible/reasonable.


Algorithms for down sampling an image? Bicubic sampling is pretty good for up sampling an image and making it larger,  but is it a good choice for down sampling as well? Are there better choices?

When Sean and I wrote stb_image_resize we chose Mitchell for downsizing. Mitchell is similar to Cubic, you can read about the cubic class of sampling filters in Mitchell Netravali 1988. They are all pretty similar and will get you very similar results.

I can't find any written record between Sean and I of why we decided to go with Mitchell, but if memory serves we just resampled a bunch of images and used the algorithm that we thought looked best. I wouldn't say that there is one authoritative or best filter, you should use the one that looks best on your data.

Edit: Like joojaa says, a windowed sinc filter is also good, if not quite as cheap. You can find some implementations here.


CPU load in games - what is the GPU/DirectX doing to make CPU load higher? More of a question about what's possible in DX.

Some people have reported that when running games on higher settings they get less CPU load but when dropping the settings the CPU load increases.

What would be causing increased CPU load in these cases?

Someone suggested that the DirectX API falls back to software when certain features are turned off, but is this something that can happen - can DirectX do a mixed render where some stuff in the render pipeline is pushed through the CPU and the rest goes to the GPU?

My thoughts were texture quality and downsampling and potentially physics processing not being offloaded but does anyone actually know what the cause might be?

The most likely explanation is that the game is GPU-bounded and not running at a fixed frame rate. If reducing the quality in the settings allows the game to reach a higher frame rate, the GPU load will be the same, but the CPU load will increase: the CPU is doing the same amount of work per frame (or possibly less), but there are more frames per second, so the total amount of CPU work per second increases. If the game does some non-graphics work per frame as well (for example, if physics updates are at the same rate, or if input is polled once per frame), the effect will be even greater.

Forcing rendering to a fixed frame-rate would allow a controlled comparison.

Your hypothesis about software emulation of features may apply to some features in some games. A particular hardware feature might be exposed as an option because it's slow on some GPUs, and turning the feature off uses a CPU implementation in the game instead. If you have more specific questions about that, be sure to ask a new question which includes the source code you're working on and the benchmark results.


What factors affect which shader language to learn? I don't know any shader languages. I've heard of GLSL and HLSL, and I'm interested in learning one or both.

Are there significant differences between them that would make one or other better in certain situations? Is it useful to know both or would either cover most needs?

I don't want vague answers indicating personal preference. I'm looking for specific measurable differences so that I can decide for myself which will suit me best. I don't have a specific task in mind - I'm hoping to discover whether there is one or other that I can learn and then apply to any future tasks, rather than having to learn a new language for each new task.

If there are other shader languages which I have not mentioned I would be interested to hear the comparison for those too, provided they are not dependent on any particular GPU manufacturer. I want my code to be portable across different graphics cards.

The shader language is bound to the APIs/engines that support it (glsl to openGL & WebGL and hlsl to D3D). There are tools to translate from one to the other but they aren't perfect.

This means the main reason to pick one over the other is which platform you will be working on.

However by the end of the year a new binary shader representation will come out called Spir-V designed for Vulkan and an OpenGL extension for it is expected to follow shortly. This will allow you to write shaders in many languages as long as a translation tool to Spir-V exists (several are in the works including a Python and Haskell tool).


Does normal mapping make sense for a view of earth from space? I am working on a planet visualizer, which at this point is little more than the NASA blue marble image applied as a diffuse texture.  For the atmospheric rim, I am simply blending in a constant color depending on Fresnel.  It looks like this:

Besides the obvious problems (no clouds) I am struggling to understand what techniques are applicable at this scale.  As I understand it, normal mapping is simply a modeling of surface features below the threshold of polygon resolution.  Likewise, roughness-based BRDFs are simply a modeling of surface features below the threshold of normal maps.  In my case, I am using the 8k texture map at 500m resolution.  Does it make sense to normal map the Earth from this distance?  How would I even do that?  Is there a height map for the planet?

Are there other techniques that are more applicable to rendering (real) planets like this?  My next step is to make a specular mask based on water cover, but I feel like I am missing something else.

P.S.  I noticed that the NASA blue marble image didn't render well when I enabled gamma correction, nor when I divided its texel values by $\pi$ before computing the light value.  I'm not sure why that is.

In my opinion, yes, it does. I've heard it claimed that in reality, the ratio of the tallest mountain on Earth to the Earth's diameter is smaller than the variations in height of an apple's skin to its diameter. So I can't say that it would be realistic, but it certainly looks better than not doing it. At the least it gives you some shading between high and low spots, even with a simple directional light.


Screen capture of game video I'd like to "hook" into a running game, say Mario Bros, and capture each frame rendered... saving that frame to an image file. A good example of something similar is FRAPS. --Note: I don't want to capture the whole screen/desktop. I want to capture a targeted window.

I have had a look at OBS (Open Broadcasting Software) but it is not particularly fast. Don't get me wrong, it's great software, but unfortunately there is no/poor documentation, making a massive project written in c and c++ nearly inaccessible for a new-to-c++ programmer.

I've also had a look at GamingAnywhere, but unfortunately, I am unable to get it to work, there is very little/no documentation, runs in VS2010 only and is messy (with poor variable naming). However, it is a research project and so it is understandably undocumented and messy.

I know that this can be done with OpenGL, GDI and with Direct3D, but I am unable to find some good examples on the net.

I read that glReadlPixels (using OpenGL) can be used and I've read the documentation, but the post mentioned nothing about hooking into a running game/application graphics.

Questions:

Can I hook into the graphics of a game developed with OpenGL, using,
say, Direct3D? Does the library used for hooking have to be the same
as the one used by the game?
How do I hook into the game's rendered frames so that I can output
those frames to image files or to a video file? (Just some links or
brief explanation of what I need to do would be great)
BackBuffer - I read that it is very fast accessing the BackBuffer to
retrieve the frames. Does someone have an example for me on how to
do this with the latest libraries? I have found that most examples
are out of date.
For my purposes, is there any clearly "this is faster than that"? What I mean is, would, say, OpenGL, be faster for my purposes?

If someone knows of an open-source project (that does essentially what I need) that is actively developed and well documented, I'd love to know about it.

It is possible to do what you describe, but I'm afraid it is not a trivial process. Actually, this will be very tied to the Operating Systems you are targeting and possibly also requiring specific tweaks for the given game/application.

I would start looking into DLL injection techniques, which should allow, for instance, intercepting calls to the Direct3D (Windows) or OpenGL APIs, which you can then use to copy de framebuffers of the application. A quick Internet search turns up this and this with detailed explanation.

I once wrote a small-scale OpenGL interceptor on MacOS by injecting a custom shared module, but it was under very controlled circumstances and making a lot of assumptions, such as having root privileges in the system, so for a production tool, it would be a lot more complicated. You can find another very interesting project here about intercepting the D3D API to install an automated AI bot on Starcraft, which should touch on very similar concepts to what you are looking for.

Once you manage to intercept the underlying rendering API, you could then perhaps hook your code into each SwapBuffers or equivalent call to just copy the previous framebuffer before the swap, then save it (this is where something like glReadPixels would come into play).

Copying and saving the framebuffer in an efficient manner is also challenging. The easy way is to just dump each frame as an uncompressed RGB image, but then you'll wind up with hundreds of gigabytes of data for just a couple of minutes of gameplay, so unless you have a nice HDD array sitting at the corner of your table ;), you'll need to look into compressing the frames somehow.

The downside of compressing the frames now is that it takes a lot of processing, so a naive approach might turn the once interactive application you are trying to record into an interactive slide-show :(. So at this stage you'd have to start thinking about optimizations.

Don't know of any projects out there providing a library for efficient frame capture for games, but this would certainly be something nice to have! I think one thing that might be holding such project back is, like I mentioned at the beginning, that this is a very system dependent thing, so cases of use will most likely be limited to a single platform or OS.


I'm trying to get a 2D screen position so I unproject a point from 3D into 2D screen coordinates by that following function: I'm trying to get a 2D screen position so I unproject a point from 3D into 2D screen coordinates by that following function:

Ogre::Vector2 SpaceTransformHelper::convertWorldToScreenPosition( const Ogre::Vector3& worldPosition )
{
    Stages::AStage* currStage = Stages::StageManager::getSingleton()->getActiveStage();
    Core::Resolution screenRes = Core::EnvironmentInformation::getSingleton()->getScreenResolution();

    // Get worldviewprojection from camera.
    Ogre::Matrix4 worldViewProjection = currStage->getActiveCamera()->getCameraViewProjectionMatrix();

    // Transform world coordinates.
    Ogre::Vector3 transformedWorldPosition = worldViewProjection * worldPosition;

    // Calculate screen coordinates from world coordinates.
    Ogre::Vector2 screenPosition;
    screenPosition.x = (int) ( ( ( transformedWorldPosition.x + 1 ) / 2.0 ) * screenRes.width + 0.5 );
    screenPosition.y = (int) ( ( ( 1 - transformedWorldPosition.y ) / 2.0 ) * screenRes.height + 0.5 );

    return screenPosition;
}

Now I want to convert that screenPosition into Image space to crop an image:

Core::Resolution screenRes = Core::EnvironmentInformation::getSingleton()->getScreenResolution();
float screenPositionX = (screenPosition.x / screenRes.width) * (pixelBuffer->getWidth());
float screenPositionY =  (screenPosition.y / screenRes.height) * (pixelBuffer->getHeight());

When I click on the mobile screen on the Green LED I get that cropped image.. but what I want is to only crop that green led.. Is there a problem with the transformation ?

  const Core::Resolution cameraRes = Core::EnvironmentInformation::getSingleton()->getCameraResolutionUpLoad( plane );
    Ogre::HardwarePixelBufferSharedPtr pixelBuffer = mTexture[plane]->getBuffer( 0, 0 );

    viewarassert( screenPosition.x < cameraRes.width || screenPosition.x >= 0 || screenPosition.y < cameraRes.height || screenPosition.y >= 0 );

   Core::Resolution screenRes = Core::EnvironmentInformation::getSingleton()->getScreenResolution();
   std::pair< Ogre::Vector2, Ogre::Vector2 > uvBounds = calculateSingleViewTextureCoordinates();

    float screenPositionX = cameraRes.height - (screenPosition.x / screenRes.width) * ((int)pixelBuffer->getHeight());
    float screenPositionY=  cameraRes.width -  (screenPosition.y / screenRes.height) * ((int)pixelBuffer->getWidth());

    int  min_x = std::max( 0,(int)screenPositionX - radius);
    int  min_y = std::max( 0,(int)screenPositionY - radius);
    int  max_x = std::min( (int)cameraRes.width, (int) screenPositionX + radius );
    int  max_y = std::min( (int)cameraRes.height, (int) screenPositionY+ radius );

    // Get the pixelbox within the radius.
    pixelBuffer->lock(Ogre::HardwareBuffer::HBL_READ_ONLY);
    const Ogre::PixelBox &readrefpb = pixelBuffer->getCurrentLock();

    unsigned char *readrefdata = static_cast<unsigned char*>(readrefpb.data);

    Ogre::Image img;
    img = img.loadDynamicImage (readrefdata, pixelBuffer->getWidth(),
                                pixelBuffer->getHeight(), pixelBuffer->getFormat());
    img = img.flipAroundY();

    Ogre::Image *out = cropImage(img,  min_x, min_y,  max_x,  max_y);

Not sure if Ogre does it implicitely, but using a 4x4 matrix to transform a 3D point to get the correct 2D coordinates (as it would be rendered with the same matrix),  requires a transition from homogeneous coordinates by dividing the result by w.

In other words, you must multiply the matrix by a 4-component vector (x,y,z,1.0f) instead of a 3-component vector. And then the resulting vector will look like (X,Y,Z,W) and you need to divide by W to get the position like it would be rendered.

Like this:

// Transform world coordinates.
Ogre::Vector4 transformedWorldPosition = worldViewProjection * Ogre::Vector4(worldPosition,1.0);

//Divide by w
transformedWorldPosition = transformedWorldPosition / transformedWorldPosition.w;

// Calculate screen coordinates from world coordinates.
Ogre::Vector2 screenPosition;
screenPosition.x = (int) ( ( ( transformedWorldPosition.x + 1 ) / 2.0 ) * screenRes.width + 0.5 );
screenPosition.y = (int) ( ( ( 1 - transformedWorldPosition.y ) / 2.0 ) * screenRes.height + 0.5 );


How am I able to perform perspective projection without a near plane? I'm learning about 3D computer graphics, but I'm having a hard time understanding why the near plane of a viewing frustum can not be placed at z position $0$ (right at the camera).

I can understand conceptually that the near plane is essentially the retinal canvas -- so by definition it must exist to some extent -- but I have trouble understanding why the near plane isn't simply an abstract concept where it's actual position is infinitesimally close to the camera position, as opposed to a definite z position away.

In several explanations, the following formula is given to explain why $0$ cannot be used as the near plan position (where $A$ is the camera position, $B$ is an object vertex, and $D$ is the perspective projection of point $B$ onto the near plane):

$${ BC \over AC } = { DE \over AE }$$

In this case, the geometry of similar triangles $ABC$ and $ADE$ is used to determine the height of $D$ via the solution of $DE$. It is obvious that if the near plane is at $0$ ($AE = 0$), then a division by $0$ occurs -- hence, why the near plane cannot be located at position $0$.

However, why is this method used to determine the position of $D$ on the canvas?

I've written a simple raycasting visualizer before and didn't have an explicitly defined near plane. In my engine, I simply defined a $60^\circ$ field of view and divided the number pixels on my screen among that field of view. For example, for a $300$x$300$ screen:

$$1\text{ pixel} = 300/60^\circ = 5^\circ$$

Next, I found the angle between my camera and the object vertex ($\angle BAC$) and divided it by $5^\circ$ to acquire the pixel coordinate on my screen. In this method, no explicit near plane was necessary and I used my actual camera position to determine the angle.

So how was I able to perform a perspective projection without a near plane in my raycasting method?

The near and far planes of a viewing frustum aren't needed for simple 3D→2D projection. What the near and far planes actually do, in a typical rasterizer setup, is define the range of values for the depth buffer.

Depths in the [near, far] range will be mapped into [0, 1] to be stored in the depth buffer. However, the depths aren't simply linearly rescaled. Instead, we use a hyperbolic mapping for depth: the value stored in the depth buffer is the reciprocal of the actual depth distance, with a scale and offset applied.

If you look at this curve and imagine moving the near plane value toward $z = 0$, the corresponding point on the $1/z$ curve would shoot up toward infinity. The math blows up and the depth buffer becomes useless (it suffers a catastrophic loss of precision).

The reason why we use the reciprocal instead of some other function of depth is basically convenience: the reciprocal depth interpolates nicely in screen space, and it fits naturally into the mathematical framework of perspective projections: there's already a divide by $z$ being applied to the $x, y$ coordinates.

If you'd like to know more, I have a short article on the topic: Depth Precision Visualized.