## [Basic syntax](#Basic-syntax)
The (basic) syntax of `@nloops` is as follows:
-   The first argument must be an integer (*not* a variable) specifying the number of loops.
-   The second argument is the symbol-prefix used for the iterator variable. Here we used `i`, and variables `i_1, i_2, i_3` were generated.
-   The third argument specifies the range for each iterator variable. If you use a variable (symbol) here, it's taken as `axes(A, dim)`. More flexibly, you can use the anonymous-function expression syntax described below.
-   The last argument is the body of the loop. Here, that's what appears between the `begin...end`.
There are some additional features of `@nloops` described in the [reference section](#dev-cartesian-reference).
`@nref` follows a similar pattern, generating `A[i_1,i_2,i_3]` from `@nref 3 A i`. The general practice is to read from left to right, which is why `@nloops` is `@nloops 3 i A expr` (as in `for i_2 = axes(A, 2)`, where `i_2` is to the left and the range is to the right) whereas `@nref` is `@nref 3 A i` (as in `A[i_1,i_2,i_3]`, where the array comes first).
If you're developing code with Cartesian, you may find that debugging is easier when you examine the generated code, using `@macroexpand`:
```julia-repl
julia> @macroexpand @nref 2 A i
:(A[i_1, i_2])
```


### [Supplying the number of expressions](#Supplying-the-number-of-expressions)
The first argument to both of these macros is the number of expressions, which must be an integer. When you're writing a function that you intend to work in multiple dimensions, this may not be something you want to hard-code. The recommended approach is to use a `@generated function`. Here's an example:
```julia
@generated function mysum(A::Array{T,N}) where {T,N}
    quote
        s = zero(T)
        @nloops $N i A begin
            s += @nref $N A i
        end
        s
    end
end
```
Naturally, you can also prepare expressions or perform calculations before the `quote` block.


### [Anonymous-function expressions as macro arguments](#Anonymous-function-expressions-as-macro-arguments)
Perhaps the single most powerful feature in `Cartesian` is the ability to supply anonymous-function expressions that get evaluated at parsing time. Let's consider a simple example:
```julia
@nexprs 2 j->(i_j = 1)
```
`@nexprs` generates `n` expressions that follow a pattern. This code would generate the following statements:
```julia
i_1 = 1
i_2 = 1
```
In each generated statement, an "isolated" `j` (the variable of the anonymous function) gets replaced by values in the range `1:2`. Generally speaking, Cartesian employs a LaTeX-like syntax. This allows you to do math on the index `j`. Here's an example computing the strides of an array:
```julia
s_1 = 1
@nexprs 3 j->(s_{j+1} = s_j * size(A, j))
```
would generate expressions
```julia
s_1 = 1
s_2 = s_1 * size(A, 1)
s_3 = s_2 * size(A, 2)
s_4 = s_3 * size(A, 3)
```
Anonymous-function expressions have many uses in practice.


#### [Macro reference](#dev-cartesian-reference)
```julia
@nloops N itersym rangeexpr bodyexpr
@nloops N itersym rangeexpr preexpr bodyexpr
@nloops N itersym rangeexpr preexpr postexpr bodyexpr
```
Generate `N` nested loops, using `itersym` as the prefix for the iteration variables. `rangeexpr` may be an anonymous-function expression, or a simple symbol `var` in which case the range is `axes(var, d)` for dimension `d`.
Optionally, you can provide "pre" and "post" expressions. These get executed first and last, respectively, in the body of each loop. For example:
```julia
@nloops 2 i A d -> j_d = min(i_d, 5) begin
    s += @nref 2 A j
end
```
would generate:
```julia
for i_2 = axes(A, 2)
    j_2 = min(i_2, 5)
    for i_1 = axes(A, 1)
        j_1 = min(i_1, 5)
        s += A[j_1, j_2]
    end
end
```
If you want just a post-expression, supply [`nothing`](../../base/constants/#Core.nothing) for the pre-expression. Using parentheses and semicolons, you can supply multi-statement expressions.
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/cartesian.jl#L9-L37)
```julia
@nref N A indexexpr
```
Generate expressions like `A[i_1, i_2, ...]`. `indexexpr` can either be an iteration-symbol prefix, or an anonymous-function expression.
**Examples**
```julia-repl
julia> @macroexpand Base.Cartesian.@nref 3 A i
:(A[i_1, i_2, i_3])
```
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/cartesian.jl#L72-L83)
```julia
@nextract N esym isym
```


### [Anonymous-function expressions as macro arguments](#Anonymous-function-expressions-as-macro-arguments)
Generate `N` variables `esym_1`, `esym_2`, ..., `esym_N` to extract values from `isym`. `isym` can be either a `Symbol` or anonymous-function expression.
`@nextract 2 x y` would generate
```julia
x_1 = y[1]
x_2 = y[2]
```
while `@nextract 3 x d->y[2d-1]` yields
```julia
x_1 = y[1]
x_2 = y[3]
x_3 = y[5]
```
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/cartesian.jl#L160-L177)
```julia
@nexprs N expr
```
Generate `N` expressions. `expr` should be an anonymous-function expression.
**Examples**
```julia-repl
julia> @macroexpand Base.Cartesian.@nexprs 4 i -> y[i] = A[i+j]
quote
    y[1] = A[1 + j]
    y[2] = A[2 + j]
    y[3] = A[3 + j]
    y[4] = A[4 + j]
end
```
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/cartesian.jl#L139-L154)
```julia
@ncall N f sym...
```
Generate a function call expression. `sym` represents any number of function arguments, the last of which may be an anonymous-function expression and is expanded into `N` arguments.
For example, `@ncall 3 func a` generates
```julia
func(a_1, a_2, a_3)
```
while `@ncall 2 func a b i->c[i]` yields
```julia
func(a, b, c[1], c[2])
```
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/cartesian.jl#L89-L103)
```julia
@ncallkw N f kw sym...
```


### [Anonymous-function expressions as macro arguments](#Anonymous-function-expressions-as-macro-arguments)
Generate a function call expression with keyword arguments `kw...`. As in the case of [`@ncall`](#Base.Cartesian.@ncall), `sym` represents any number of function arguments, the last of which may be an anonymous-function expression and is expanded into `N` arguments.
**Examples**
```julia-repl
julia> using Base.Cartesian

julia> f(x...; a, b = 1, c = 2, d = 3) = +(x..., a, b, c, d);

julia> x_1, x_2 = (-1, -2); b = 0; kw = (c = 0, d = 0);

julia> @ncallkw 2 f (; a = 0, b, kw...) x
-3
```
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/cartesian.jl#L111-L130)
```julia
@ntuple N expr
```
Generates an `N`-tuple. `@ntuple 2 i` would generate `(i_1, i_2)`, and `@ntuple 2 k->k+1` would generate `(2,3)`.
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/cartesian.jl#L221-L226)
```julia
@nall N expr
```
Check whether all of the expressions generated by the anonymous-function expression `expr` evaluate to `true`.
`@nall 3 d->(i_d > 1)` would generate the expression `(i_1 > 1 && i_2 > 1 && i_3 > 1)`. This can be convenient for bounds-checking.
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/cartesian.jl#L188-L196)
```julia
@nany N expr
```
Check whether any of the expressions generated by the anonymous-function expression `expr` evaluate to `true`.
`@nany 3 d->(i_d > 1)` would generate the expression `(i_1 > 1 || i_2 > 1 || i_3 > 1)`.


### [Anonymous-function expressions as macro arguments](#Anonymous-function-expressions-as-macro-arguments)
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/cartesian.jl#L205-L212)
```julia
@nif N conditionexpr expr
@nif N conditionexpr expr elseexpr
```
Generates a sequence of `if ... elseif ... else ... end` statements. For example:
```julia
@nif 3 d->(i_d >= size(A,d)) d->(error("Dimension ", d, " too big")) d->println("All OK")
```
would generate:
```julia
if i_1 > size(A, 1)
    error("Dimension ", 1, " too big")
elseif i_2 > size(A, 2)
    error("Dimension ", 2, " too big")
else
    println("All OK")
end
```
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/cartesian.jl#L232-L249)
------------------------------------------------------------------------


# Talking to the compiler (the :meta mechanism) · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/meta/


# [Talking to the compiler (the :meta mechanism)](#Talking-to-the-compiler-(the-:meta-mechanism))
In some circumstances, one might wish to provide hints or instructions that a given block of code has special properties: you might always want to inline it, or you might want to turn on special compiler optimization passes. Starting with version 0.4, Julia has a convention that these instructions can be placed inside a `:meta` expression, which is typically (but not necessarily) the first expression in the body of a function.
`:meta` expressions are created with macros. As an example, consider the implementation of the `@inline` macro:
```julia
macro inline(ex)
    esc(isa(ex, Expr) ? pushmeta!(ex, :inline) : ex)
end
```
Here, `ex` is expected to be an expression defining a function. A statement like this:
```julia
@inline function myfunction(x)
    x*(x+3)
end
```
gets turned into an expression like this:
```julia
quote
    function myfunction(x)
        Expr(:meta, :inline)
        x*(x+3)
    end
end
```
`Base.pushmeta!(ex, tag::Union{Symbol,Expr})` appends `:tag` to the end of the `:meta` expression, creating a new `:meta` expression if necessary.


# [Talking to the compiler (the :meta mechanism)](#Talking-to-the-compiler-(the-:meta-mechanism))
To use the metadata, you have to parse these `:meta` expressions. If your implementation can be performed within Julia, `Base.popmeta!` is very handy: `Base.popmeta!(body, :symbol)` will scan a function *body* expression (one without the function signature) for the first `:meta` expression containing `:symbol`, extract any arguments, and return a tuple `(found::Bool, args::Array{Any})`. If the metadata did not have any arguments, or `:symbol` was not found, the `args` array will be empty.
Not yet provided is a convenient infrastructure for parsing `:meta` expressions from C++.
------------------------------------------------------------------------


# SubArrays · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/subarrays/


# [SubArrays](#SubArrays)
Julia's `SubArray` type is a container encoding a "view" of a parent [`AbstractArray`](../../base/arrays/#Core.AbstractArray). This page documents some of the design principles and implementation of `SubArray`s.
One of the major design goals is to ensure high performance for views of both [`IndexLinear`](../../base/arrays/#Base.IndexLinear) and [`IndexCartesian`](../../base/arrays/#Base.IndexCartesian) arrays. Furthermore, views of `IndexLinear` arrays should themselves be `IndexLinear` to the extent that it is possible.


## [Index replacement](#Index-replacement)
Consider making 2d slices of a 3d array:
```julia-repl
julia> A = rand(2,3,4);

julia> S1 = view(A, :, 1, 2:3)
2×2 view(::Array{Float64, 3}, :, 1, 2:3) with eltype Float64:
 0.839622  0.711389
 0.967143  0.103929

julia> S2 = view(A, 1, :, 2:3)
3×2 view(::Array{Float64, 3}, 1, :, 2:3) with eltype Float64:
 0.839622  0.711389
 0.789764  0.806704
 0.566704  0.962715
```
`view` drops "singleton" dimensions (ones that are specified by an `Int`), so both `S1` and `S2` are two-dimensional `SubArray`s. Consequently, the natural way to index these is with `S1[i,j]`. To extract the value from the parent array `A`, the natural approach is to replace `S1[i,j]` with `A[i,1,(2:3)[j]]` and `S2[i,j]` with `A[1,i,(2:3)[j]]`.
The key feature of the design of SubArrays is that this index replacement can be performed without any runtime overhead.


## [SubArray design](#SubArray-design)


### [Type parameters and fields](#Type-parameters-and-fields)
The strategy adopted is first and foremost expressed in the definition of the type:
```julia
struct SubArray{T,N,P,I,L} <: AbstractArray{T,N}
    parent::P
    indices::I
    offset1::Int       # for linear indexing and pointer, only valid when L==true
    stride1::Int       # used only for linear indexing
    ...
end
```
`SubArray` has 5 type parameters. The first two are the standard element type and dimensionality. The next is the type of the parent `AbstractArray`. The most heavily-used is the fourth parameter, a `Tuple` of the types of the indices for each dimension. The final one, `L`, is only provided as a convenience for dispatch; it's a boolean that represents whether the index types support fast linear indexing. More on that later.
If in our example above `A` is a `Array{Float64, 3}`, our `S1` case above would be a `SubArray{Float64,2,Array{Float64,3},Tuple{Base.Slice{Base.OneTo{Int64}},Int64,UnitRange{Int64}},false}`. Note in particular the tuple parameter, which stores the types of the indices used to create `S1`. Likewise,
```julia-repl
julia> S1.indices
(Base.Slice(Base.OneTo(2)), 1, 2:3)
```
Storing these values allows index replacement, and having the types encoded as parameters allows one to dispatch to efficient algorithms.


### [Index translation](#Index-translation)
Performing index translation requires that you do different things for different concrete `SubArray` types. For example, for `S1`, one needs to apply the `i,j` indices to the first and third dimensions of the parent array, whereas for `S2` one needs to apply them to the second and third. The simplest approach to indexing would be to do the type-analysis at runtime:
```julia
parentindices = Vector{Any}()
for thisindex in S.indices
    ...
    if isa(thisindex, Int)
        # Don't consume one of the input indices
        push!(parentindices, thisindex)
    elseif isa(thisindex, AbstractVector)
        # Consume an input index
        push!(parentindices, thisindex[inputindex[j]])
        j += 1
    elseif isa(thisindex, AbstractMatrix)
        # Consume two input indices
        push!(parentindices, thisindex[inputindex[j], inputindex[j+1]])
        j += 2
    elseif ...
end
S.parent[parentindices...]
```
Unfortunately, this would be disastrous in terms of performance: each element access would allocate memory, and involves the running of a lot of poorly-typed code.
The better approach is to dispatch to specific methods to handle each type of stored index. That's what `reindex` does: it dispatches on the type of the first stored index and consumes the appropriate number of input indices, and then it recurses on the remaining indices. In the case of `S1`, this expands to
```julia
Base.reindex(S1, S1.indices, (i, j)) == (i, S1.indices[2], S1.indices[3][j])
```


### [Index translation](#Index-translation)
for any pair of indices `(i,j)` (except [`CartesianIndex`](../../base/arrays/#Base.IteratorsMD.CartesianIndex)s and arrays thereof, see below).
This is the core of a `SubArray`; indexing methods depend upon `reindex` to do this index translation. Sometimes, though, we can avoid the indirection and make it even faster.


### [Linear indexing](#Linear-indexing)
Linear indexing can be implemented efficiently when the entire array has a single stride that separates successive elements, starting from some offset. This means that we can pre-compute these values and represent linear indexing simply as an addition and multiplication, avoiding the indirection of `reindex` and (more importantly) the slow computation of the cartesian coordinates entirely.
For `SubArray` types, the availability of efficient linear indexing is based purely on the types of the indices, and does not depend on values like the size of the parent array. You can ask whether a given set of indices supports fast linear indexing with the internal `Base.viewindexing` function:
```julia-repl
julia> Base.viewindexing(S1.indices)
IndexCartesian()

julia> Base.viewindexing(S2.indices)
IndexLinear()
```
This is computed during construction of the `SubArray` and stored in the `L` type parameter as a boolean that encodes fast linear indexing support. While not strictly necessary, it means that we can define dispatch directly on `SubArray{T,N,A,I,true}` without any intermediaries.
Since this computation doesn't depend on runtime values, it can miss some cases in which the stride happens to be uniform:
```julia-repl
julia> A = reshape(1:4*2, 4, 2)
4×2 reshape(::UnitRange{Int64}, 4, 2) with eltype Int64:
 1  5
 2  6
 3  7
 4  8

julia> diff(A[2:2:4,:][:])
3-element Vector{Int64}:
 2
 2
 2
```


### [Linear indexing](#Linear-indexing)
A view constructed as `view(A, 2:2:4, :)` happens to have uniform stride, and therefore linear indexing indeed could be performed efficiently. However, success in this case depends on the size of the array: if the first dimension instead were odd,
```julia-repl
julia> A = reshape(1:5*2, 5, 2)
5×2 reshape(::UnitRange{Int64}, 5, 2) with eltype Int64:
 1   6
 2   7
 3   8
 4   9
 5  10

julia> diff(A[2:2:4,:][:])
3-element Vector{Int64}:
 2
 3
 2
```
then `A[2:2:4,:]` does not have uniform stride, so we cannot guarantee efficient linear indexing. Since we have to base this decision based purely on types encoded in the parameters of the `SubArray`, `S = view(A, 2:2:4, :)` cannot implement efficient linear indexing.


### [A few details](#A-few-details)
-   Note that the `Base.reindex` function is agnostic to the types of the input indices; it simply determines how and where the stored indices should be reindexed. It not only supports integer indices, but it supports non-scalar indexing, too. This means that views of views don't need two levels of indirection; they can simply re-compute the indices into the original parent array!
-   Hopefully by now it's fairly clear that supporting slices means that the dimensionality, given by the parameter `N`, is not necessarily equal to the dimensionality of the parent array or the length of the `indices` tuple. Neither do user-supplied indices necessarily line up with entries in the `indices` tuple (e.g., the second user-supplied index might correspond to the third dimension of the parent array, and the third element in the `indices` tuple).
    What might be less obvious is that the dimensionality of the stored parent array must be equal to the number of effective indices in the `indices` tuple. Some examples:
    ```julia
    A = reshape(1:35, 5, 7) # A 2d parent Array
    S = view(A, 2:7)         # A 1d view created by linear indexing
    S = view(A, :, :, 1:1)   # Appending extra indices is supported
    ```


### [A few details](#A-few-details)
    Naively, you'd think you could just set `S.parent = A` and `S.indices = (:,:,1:1)`, but supporting this dramatically complicates the reindexing process, especially for views of views. Not only do you need to dispatch on the types of the stored indices, but you need to examine whether a given index is the final one and "merge" any remaining stored indices together. This is not an easy task, and even worse: it's slow since it implicitly depends upon linear indexing.
    Fortunately, this is precisely the computation that `ReshapedArray` performs, and it does so linearly if possible. Consequently, `view` ensures that the parent array is the appropriate dimensionality for the given indices by reshaping it if needed. The inner `SubArray` constructor ensures that this invariant is satisfied.
-   [`CartesianIndex`](../../base/arrays/#Base.IteratorsMD.CartesianIndex) and arrays thereof throw a nasty wrench into the `reindex` scheme. Recall that `reindex` simply dispatches on the type of the stored indices in order to determine how many passed indices should be used and where they should go. But with `CartesianIndex`, there's no longer a one-to-one correspondence between the number of passed arguments and the number of dimensions that they index into. If we return to the above example of `Base.reindex(S1, S1.indices, (i, j))`, you can see that the expansion is incorrect for `i, j = CartesianIndex(), CartesianIndex(2,1)`. It should *skip* the `CartesianIndex()` entirely and return:


### [A few details](#A-few-details)
    ```julia
    (CartesianIndex(2,1)[1], S1.indices[2], S1.indices[3][CartesianIndex(2,1)[2]])
    ```
    Instead, though, we get:
    ```julia
    (CartesianIndex(), S1.indices[2], S1.indices[3][CartesianIndex(2,1)])
    ```
    Doing this correctly would require *combined* dispatch on both the stored and passed indices across all combinations of dimensionalities in an intractable manner. As such, `reindex` must never be called with `CartesianIndex` indices. Fortunately, the scalar case is easily handled by first flattening the `CartesianIndex` arguments to plain integers. Arrays of `CartesianIndex`, however, cannot be split apart into orthogonal pieces so easily. Before attempting to use `reindex`, `view` must ensure that there are no arrays of `CartesianIndex` in the argument list. If there are, it can simply "punt" by avoiding the `reindex` calculation entirely, constructing a nested `SubArray` with two levels of indirection instead.
------------------------------------------------------------------------


# isbits Union Optimizations · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/isbitsunionarrays/


# [isbits Union Optimizations](#isbits-Union-Optimizations)
In Julia, the `Array` type holds both "bits" values as well as heap-allocated "boxed" values. The distinction is whether the value itself is stored inline (in the direct allocated memory of the array), or if the memory of the array is simply a collection of pointers to objects allocated elsewhere. In terms of performance, accessing values inline is clearly an advantage over having to follow a pointer to the actual value. The definition of "isbits" generally means any Julia type with a fixed, determinate size, meaning no "pointer" fields, see `?isbitstype`.
Julia also supports Union types, quite literally the union of a set of types. Custom Union type definitions can be extremely handy for applications wishing to "cut across" the nominal type system (i.e. explicit subtype relationships) and define methods or functionality on these, otherwise unrelated, set of types. A compiler challenge, however, is in determining how to treat these Union types. The naive approach (and indeed, what Julia itself did pre-0.7), is to simply make a "box" and then a pointer in the box to the actual value, similar to the previously mentioned "boxed" values. This is unfortunate, however, because of the number of small, primitive "bits" types (think `UInt8`, `Int32`, `Float64`, etc.) that would easily fit themselves inline in this "box" without needing any indirection for value access. There are two main ways Julia can take advantage of this optimization as of 0.7: isbits Union fields in types, and isbits Union Arrays.


## [isbits Union Structs](#isbits-Union-Structs)
Julia now includes an optimization wherein "isbits Union" fields in types (`mutable struct`, `struct`, etc.) will be stored inline. This is accomplished by determining the "inline size" of the Union type (e.g. `Union{UInt8, Int16}` will have a size of two bytes, which represents the size needed of the largest Union type `Int16`), and in addition, allocating an extra "type tag byte" (`UInt8`), whose value signals the type of the actual value stored inline of the "Union bytes". The type tag byte value is the index of the actual value's type in the Union type's order of types. For example, a type tag value of `0x02` for a field with type `Union{Nothing, UInt8, Int16}` would indicate that an `Int16` value is stored in the 16 bits of the field in the structure's memory; a `0x01` value would indicate that a `UInt8` value was stored in the first 8 bits of the 16 bits of the field's memory. Lastly, a value of `0x00` signals that the `nothing` value will be returned for this field, even though, as a singleton type with a single type instance, it technically has a size of 0. The type tag byte for a type's Union field is stored directly after the field's computed Union memory.


## [isbits Union Memory](#isbits-Union-Memory)
Julia can now also store "isbits Union" values inline in a Memory, as opposed to requiring an indirection box. The optimization is accomplished by storing an extra "type tag memory" of bytes, one byte per element, alongside the bytes of the actual data. This type tag memory serves the same function as the type field case: its value signals the type of the actual stored Union value. The "type tag memory" directly follows the regular data space. So the formula to access an isbits Union Array's type tag bytes is `a->data + a->length * a->elsize`.
------------------------------------------------------------------------


# System Image Building · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/sysimg/


# [System Image Building](#System-Image-Building)


## [Building the Julia system image](#Building-the-Julia-system-image)
Julia ships with a preparsed system image containing the contents of the `Base` module, named `sys.ji`. This file is also precompiled into a shared library called `sys.{so,dll,dylib}` on as many platforms as possible, so as to give vastly improved startup times. On systems that do not ship with a precompiled system image file, one can be generated from the source files shipped in Julia's `DATAROOTDIR/julia/base` folder.
Julia will by default generate its system image on half of the available system threads. This may be controlled by the [`JULIA_IMAGE_THREADS`](../../manual/environment-variables/#JULIA_IMAGE_THREADS) environment variable.
This operation is useful for multiple reasons. A user may:
-   Build a precompiled shared library system image on a platform that did not ship with one, thereby improving startup times.
-   Modify `Base`, rebuild the system image and use the new `Base` next time Julia is started.
-   Include a `userimg.jl` file that includes packages into the system image, thereby creating a system image that has packages embedded into the startup environment.
The [`PackageCompiler.jl` package](https://github.com/JuliaLang/PackageCompiler.jl) contains convenient wrapper functions to automate this process.


## [System image optimized for multiple microarchitectures](#sysimg-multi-versioning)
The system image can be compiled simultaneously for multiple CPU microarchitectures under the same instruction set architecture (ISA). Multiple versions of the same function may be created with minimum dispatch point inserted into shared functions in order to take advantage of different ISA extensions or other microarchitecture features. The version that offers the best performance will be selected automatically at runtime based on available CPU features.


### [Specifying multiple system image targets](#Specifying-multiple-system-image-targets)
A multi-microarchitecture system image can be enabled by passing multiple targets during system image compilation. This can be done either with the [`JULIA_CPU_TARGET`](../../manual/environment-variables/#JULIA_CPU_TARGET) make option or with the `-C` command line option when running the compilation command manually. Multiple targets are separated by `;` in the option string. The syntax for each target is a CPU name followed by multiple features separated by `,`. All features supported by LLVM are supported and a feature can be disabled with a `-` prefix. (`+` prefix is also allowed and ignored to be consistent with LLVM syntax). Additionally, a few special features are supported to control the function cloning behavior.
It is good practice to specify either `clone_all` or `base(<n>)` for every target apart from the first one. This makes it explicit which targets have all functions cloned, and which targets are based on other targets. If this is not done, the default behavior is to not clone every function, and to use the first target's function definition as the fallback when not cloning a function.
1.  `clone_all`
    By default, only functions that are the most likely to benefit from the microarchitecture features will be cloned. When `clone_all` is specified for a target, however, **all** functions in the system image will be cloned for the target. The negative form `-clone_all` can be used to prevent the built-in heuristic from cloning all functions.
2.  `base(<n>)`


### [Specifying multiple system image targets](#Specifying-multiple-system-image-targets)
    Where `<n>` is a placeholder for a non-negative number (e.g. `base(0)`, `base(1)`). By default, a partially cloned (i.e. not `clone_all`) target will use functions from the default target (first one specified) if a function is not cloned. This behavior can be changed by specifying a different base with the `base(<n>)` option. The `n`th target (0-based) will be used as the base target instead of the default (`0`th) one. The base target has to be either `0` or another `clone_all` target. Specifying a non-`clone_all` target as the base target will cause an error.
3.  `opt_size`
    This causes the function for the target to be optimized for size when there isn't a significant runtime performance impact. This corresponds to `-Os` GCC and Clang option.
4.  `min_size`
    This causes the function for the target to be optimized for size that might have a significant runtime performance impact. This corresponds to `-Oz` Clang option.
As an example, at the time of this writing, the following string is used in the creation of the official `x86_64` Julia binaries downloadable from julialang.org:
```julia
generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)
```


### [Specifying multiple system image targets](#Specifying-multiple-system-image-targets)
This creates a system image with three separate targets; one for a generic `x86_64` processor, one with a `sandybridge` ISA (explicitly excluding `xsaveopt`) that explicitly clones all functions, and one targeting the `haswell` ISA, based off of the `sandybridge` sysimg version, and also excluding `rdrnd`. When a Julia implementation loads the generated sysimg, it will check the host processor for matching CPU capability flags, enabling the highest ISA level possible. Note that the base level (`generic`) requires the `cx16` instruction, which is disabled in some virtualization software and must be enabled for the `generic` target to be loaded. Alternatively, a sysimg could be generated with the target `generic,-cx16` for greater compatibility, however note that this may cause performance and stability problems in some code.


### [Implementation overview](#Implementation-overview)
This is a brief overview of different part involved in the implementation. See code comments for each components for more implementation details.
1.  System image compilation
    The parsing and cloning decision are done in `src/processor*`. We currently support cloning of function based on the present of loops, simd instructions, or other math operations (e.g. fastmath, fma, muladd). This information is passed on to `src/llvm-multiversioning.cpp` which does the actual cloning. In addition to doing the cloning and insert dispatch slots (see comments in `MultiVersioning::runOnModule` for how this is done), the pass also generates metadata so that the runtime can load and initialize the system image correctly. A detailed description of the metadata is available in `src/processor.h`.
2.  System image loading
    The loading and initialization of the system image is done in `src/processor*` by parsing the metadata saved during system image generation. Host feature detection and selection decision are done in `src/processor_*.cpp` depending on the ISA. The target selection will prefer exact CPU name match, larger vector register size, and larger number of features. An overview of this process is in `src/processor.cpp`.
------------------------------------------------------------------------


# Package Images · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/pkgimg/


# [Package Images](#pkgimages)
Julia package images provide object (native code) caches for Julia packages. They are similar to Julia's [system image](../eval/#dev-sysimg) and support many of the same features. In fact the underlying serialization format is the same, and the system image is the base image that the package images are build against.


## [High-level overview](#High-level-overview)
Package images are shared libraries that contain both code and data. Like `.ji` cache files, they are generated per package. The data section contains both global data (global variables in the package) as well as the necessary metadata about what methods and types are defined by the package. The code section contains native objects that cache the final output of Julia's LLVM-based compiler.
The command line option `--pkgimages=no` can be used to turn off object caching for this session. Note that this means that cache files have to likely be regenerated. See [`JULIA_MAX_NUM_PRECOMPILE_FILES`](../../manual/environment-variables/#JULIA_MAX_NUM_PRECOMPILE_FILES) for the upper limit of variants Julia caches per default.
While the package images present themselves as native shared libraries, they are only an approximation thereof. You will not be able to link against them from a native program and they must be loaded from Julia.


## [Linking](#Linking)
Since the package images contain native code, we must run a linker over them before we can use them. You can set the environment variable [`JULIA_VERBOSE_LINKING`](../../manual/environment-variables/#JULIA_VERBOSE_LINKING) to `true` to make the package image linking process verbose.
Furthermore, we cannot assume that the user has a working system linker installed. Therefore, Julia ships with LLD, the LLVM linker, to provide a working out of the box experience. In `base/linking.jl`, we implement a limited interface to be able to link package images on all supported platforms.


### [Quirks](#Quirks)
Despite LLD being a multi-platform linker, it does not provide a consistent interface across platforms. Furthermore, it is meant to be used from `clang` or another compiler driver, we therefore reimplement some of the logic from `llvm-project/clang/lib/Driver/ToolChains`. Thankfully one can use `lld -flavor` to set lld to the right platform


#### [Windows](#Windows)
To avoid having to deal with `link.exe` we use `-flavor gnu`, effectively turning `lld` into a cross-linker from a mingw32 environment. Windows DLLs are required to contain a `_DllMainCRTStartup` function and to minimize our dependence on mingw32 libraries, we inject a stub definition ourselves.


#### [MacOS](#MacOS)
Dynamic libraries on macOS need to link against `-lSystem`. On recent macOS versions, `-lSystem` is only available for linking when Xcode is available. To that effect we link with `-undefined dynamic_lookup`.


## [Package images optimized for multiple microarchitectures](#pkgimgs-multi-versioning)
Similar to [multi-versioning](../sysimg/#sysimg-multi-versioning) for system images, package images support multi-versioning. If you are in a heterogeneous environment, with a unified cache, you can set the environment variable `JULIA_CPU_TARGET=generic` to multi-version the object caches.


## [Flags that impact package image creation and selection](#Flags-that-impact-package-image-creation-and-selection)
These are the Julia command line flags that impact cache selection. Package images that were created with different flags will be rejected.
-   `-g`, `--debug-info`: Exact match required since it changes code generation.
-   `--check-bounds`: Exact match required since it changes code generation.
-   `--inline`: Exact match required since it changes code generation.
-   `--pkgimages`: To allow running without object caching enabled.
-   `-O`, `--optimize`: Reject package images generated for a lower optimization level, but allow for higher optimization levels to be loaded.
------------------------------------------------------------------------


# Custom LLVM Passes · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/llvm-passes/


# [Custom LLVM Passes](#Custom-LLVM-Passes)
Julia has a number of custom LLVM passes. Broadly, they can be classified into passes that are required to be run to maintain Julia semantics, and passes that take advantage of Julia semantics to optimize LLVM IR.


## [Semantic Passes](#Semantic-Passes)
These passes are used to transform LLVM IR into code that is legal to be run on a CPU. Their main purpose is to enable simpler IR to be emitted by codegen, which then enables other LLVM passes to optimize common patterns.


### [CPUFeatures](#CPUFeatures)
-   Filename: `llvm-cpufeatures.cpp`
-   Class Name: `CPUFeaturesPass`
-   Opt Name: `module(CPUFeatures)`
This pass lowers the `julia.cpu.have_fma.(f32|f64)` intrinsic to either true or false, depending on the target architecture and target features present on the function. This intrinsic is often used to determine if using algorithms dependent on fast [fused multiply-add](https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_operation#Fused_multiply%E2%80%93add) operations is better than using standard algorithms not dependent on such instructions.


### [DemoteFloat16](#DemoteFloat16)
-   Filename: `llvm-demote-float16.cpp`
-   ClassName: `DemoteFloat16Pass`
-   Opt Name `function(DemoteFloat16)`
This pass replaces [float16](https://en.wikipedia.org/wiki/Half-precision_floating-point_format) operations with float32 operations on architectures that do not natively support float16 operations. This is done by inserting `fpext` and `fptrunc` instructions around any float16 operation. On architectures that do support native float16 operations, this pass is a no-op.


### [LateGCLowering](#LateGCLowering)
-   Filename: `llvm-late-gc-lowering.cpp`
-   Class Name: `LateLowerGCPass`
-   Opt Name: `function(LateLowerGCFrame)`
This pass performs most of the GC rooting work required to track pointers between GC safepoints. It also lowers several intrinsics to their corresponding instruction translation, and is permitted to violate the non-integral invariants previously established (`pointer_from_objref` is lowered to a `ptrtoint` instruction here). This pass typically occupies the most time out of all the custom Julia passes, due to its dataflow algorithm to minimize the number of objects live at any safepoint.


### [FinalGCLowering](#FinalGCLowering)
-   Filename: `llvm-final-gc-lowering.cpp`
-   Class Name: `FinalLowerGCPass`
-   Opt Name: `module(FinalLowerGC)`
This pass lowers a few last intrinsics to their final form targeting functions in the `libjulia` library. Separating this from `LateGCLowering` enables other backends (GPU compilation) to supply their own custom lowerings for these intrinsics, enabling the Julia pipeline to be used on those backends as well.


### [LowerHandlers](#LowerHandlers)
-   Filename: `llvm-lower-handlers.cpp`
-   Class Name: `LowerExcHandlersPass`
-   Opt Name: `function(LowerExcHandlers)`
This pass lowers exception handling intrinsics into calls to runtime functions that are actually called when handling exceptions.


### [RemoveNI](#RemoveNI)
-   Filename: `llvm-remove-ni.cpp`
-   Class Name: `RemoveNIPass`
-   Opt Name: `module(RemoveNI)`
This pass removes the non-integral address spaces from the module's datalayout string. This enables the backend to lower Julia's custom address spaces directly to machine code, without a costly rewrite of every pointer operation to address space 0.


### [SIMDLoop](#SIMDLoop)
-   Filename: `llvm-simdloop.cpp`
-   Class Name: `LowerSIMDLoopPass`
-   Opt Name: `loop(LowerSIMDLoop)`
This pass acts as the main driver of the `@simd` annotation. Codegen inserts a `!llvm.loopid` marker at the back branch of a loop, which this pass uses to identify loops that were originally marked with `@simd`. Then, this pass looks for a chain of floating point operations that form a reduce and adds the `contract` and `reassoc` fast math flags to allow reassociation (and thus vectorization). This pass does not preserve either loop information nor inference correctness, so it may violate Julia semantics in surprising ways. If the loop was annotated with `ivdep` as well, then the pass marks the loop as having no loop-carried dependencies (the resulting behavior is undefined if the user annotation was incorrect or gets applied to the wrong loop).


### [LowerPTLS](#LowerPTLS)
-   Filename: `llvm-ptls.cpp`
-   Class Name: `LowerPTLSPass`
-   Opt Name: `module(LowerPTLSPass)`
This pass lowers thread-local Julia intrinsics to assembly instructions. Julia relies on thread-local storage for garbage collection and multithreading task scheduling. When compiling code for system images and package images, this pass replaces calls to intrinsics with loads from global variables that are initialized at load time.
If codegen produces a function with a `swiftself` argument and calling convention, this pass assumes the `swiftself` argument is the pgcstack and will replace the intrinsics with that argument. Doing so provides speedups on architectures that have slow thread local storage accesses.


### [RemoveAddrspaces](#RemoveAddrspaces)
-   Filename: `llvm-remove-addrspaces.cpp`
-   Class Name: `RemoveAddrspacesPass`
-   Opt Name: `module(RemoveAddrspaces)`
This pass renames pointers in one address space to another address space. This is used to remove Julia-specific address spaces from LLVM IR.


### [RemoveJuliaAddrspaces](#RemoveJuliaAddrspaces)
-   Filename: `llvm-remove-addrspaces.cpp`
-   Class Name: `RemoveJuliaAddrspacesPass`
-   Opt Name: `module(RemoveJuliaAddrspaces)`
This pass removes Julia-specific address spaces from LLVM IR. It is mostly used for displaying LLVM IR in a less cluttered format. Internally, it is implemented off the RemoveAddrspaces pass.


### [Multiversioning](#Multiversioning)
-   Filename: `llvm-multiversioning.cpp`
-   Class Name: `MultiVersioningPass`
-   Opt Name: `module(JuliaMultiVersioning)`
This pass performs modifications to a module to create functions that are optimized for running on different architectures (see sysimg.md and pkgimg.md for more details). Implementation-wise, it clones functions and applies different target-specific attributes to them to allow the optimizer to use advanced features such as vectorization and instruction scheduling for that platform. It also creates some infrastructure to enable the Julia image loader to select the appropriate version of the function to call based on the architecture the loader is running on. The target-specific attributes are controlled by the `julia.mv.specs` module flag, which during compilation is derived from the [`JULIA_CPU_TARGET`](../../manual/environment-variables/#JULIA_CPU_TARGET) environment variable. The pass must also be enabled by providing a `julia.mv.enable` module flag with a value of 1.
Use of `llvmcall` with multiversioning is dangerous. `llvmcall` enables access to features not typically exposed by the Julia APIs, and are therefore usually not available on all architectures. If multiversioning is enabled and code generation is requested for a target architecture that does not support the feature required by an `llvmcall` expression, LLVM will probably error out, likely with an abort and the message `LLVM ERROR: Do not know how to split the result of this operator!`.


### [GCInvariantVerifier](#GCInvariantVerifier)
-   Filename: `llvm-gc-invariant-verifier.cpp`
-   Class Name: `GCInvariantVerifierPass`
-   Opt Name: `module(GCInvariantVerifier)`
This pass is used to verify Julia's invariants about LLVM IR. This includes things such as the nonexistence of `ptrtoint` in Julia's [non-integral address spaces](https://llvm.org/docs/LangRef.html#non-integral-pointer-type) <sup>[[nislides\]](#footnote-nislides)</sup> and the existence of only blessed `addrspacecast` instructions (Tracked -> Derived, 0 -> Tracked, etc). It performs no transformations on IR.


## [Optimization Passes](#Optimization-Passes)
These passes are used to perform transformations on LLVM IR that LLVM will not perform itself, e.g. fast math flag propagation, escape analysis, and optimizations on Julia-specific internal functions. They use knowledge about Julia's semantics to perform these optimizations.


### [CombineMulAdd](#CombineMulAdd)
-   Filename: `llvm-muladd.cpp`
-   Class Name: `CombineMulAddPass`
-   Opt Name: `function(CombineMulAdd)`
This pass serves to optimize the particular combination of a regular `fmul` with a fast `fadd` into a contract `fmul` with a fast `fadd`. This is later optimized by the backend to a [fused multiply-add](https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_operation#Fused_multiply%E2%80%93add) instruction, which can provide significantly faster operations at the cost of more [unpredictable semantics](https://simonbyrne.github.io/notes/fastmath/).
This optimization only occurs when the `fmul` has a single use, which is the fast `fadd`.


### [AllocOpt](#AllocOpt)
-   Filename: `llvm-alloc-opt.cpp`
-   Class Name: `AllocOptPass`
-   Opt Name: `function(AllocOpt)`
Julia does not have the concept of a program stack as a place to allocate mutable objects. However, allocating objects on the stack reduces GC pressure and is critical for GPU compilation. Thus, `AllocOpt` performs heap to stack conversion of objects that it can prove do not [escape](https://en.wikipedia.org/wiki/Escape_analysis) the current function. It also performs a number of other optimizations on allocations, such as removing allocations that are never used, optimizing typeof calls to freshly allocated objects, and removing stores to allocations that are immediately overwritten. The escape analysis implementation is located in `llvm-alloc-helpers.cpp`. Currently, this pass does not use information from `EscapeAnalysis.jl`, though that may change in the future.


### [PropagateJuliaAddrspaces](#PropagateJuliaAddrspaces)
-   Filename: `llvm-propagate-addrspaces.cpp`
-   Class Name: `PropagateJuliaAddrspacesPass`
-   Opt Name: `function(PropagateJuliaAddrspaces)`
This pass is used to propagate Julia-specific address spaces through operations on pointers. LLVM is not allowed to introduce or remove addrspacecast instructions by optimizations, so this pass acts to eliminate redundant addrspace casts by replacing operations with their equivalent in a Julia address space. For more information on Julia's address spaces, see (TODO link to llvm.md).


### [JuliaLICM](#JuliaLICM)
-   Filename: `llvm-julia-licm.cpp`
-   Class Name: `JuliaLICMPass`
-   Opt Name: `loop(JuliaLICM)`
This pass is used to hoist Julia-specific intrinsics out of loops. Specifically, it performs the following transformations:
1.  Hoist `gc_preserve_begin` and sink `gc_preserve_end` out of loops when the preserved objects are loop-invariant.
    1.  Since objects preserved within a loop are likely preserved for the duration of the loop, this transformation can reduce the number of `gc_preserve_begin`/`gc_preserve_end` pairs in the IR. This makes it easier for the `LateLowerGCPass` to identify where particular objects are preserved.
2.  Hoist write barriers with invariant objects
    1.  Here we assume that there are only two generations that an object can be a part of. Given that, a write barrier needs to only execute once for any pair of the same object. Thus, we can hoist write barriers out of loops when the object being written to is loop-invariant.
3.  Hoist allocations out of loops when they do not escape the loop
    1.  We use a very conservative definition of escape here, the same as the one used in `AllocOptPass`. This transformation can reduce the number of allocations in the IR, even when an allocation escapes the function altogether.


### [JuliaLICM](#JuliaLICM)
This pass is required to preserve LLVM's [MemorySSA](https://llvm.org/docs/MemorySSA.html) ([Short Video](https://www.youtube.com/watch?v=bdxWmryoHak), [Longer Video](https://www.youtube.com/watch?v=1e5y6WDbXCQ)) and [ScalarEvolution](https://baziotis.cs.illinois.edu/compilers/introduction-to-scalar-evolution.html) ([Newer Slides](https://llvm.org/devmtg/2018-04/slides/Absar-ScalarEvolution.pdf) [Older Slides](https://llvm.org/devmtg/2009-10/ScalarEvolutionAndLoopOptimization.pdf)) analyses.
-   [nislides](#citeref-nislides)https://llvm.org/devmtg/2015-02/slides/chisnall-pointers-not-int.pdf
------------------------------------------------------------------------


# Working with LLVM · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/llvm/


# [Working with LLVM](#Working-with-LLVM)
This is not a replacement for the LLVM documentation, but a collection of tips for working on LLVM for Julia.


## [Overview of Julia to LLVM Interface](#Overview-of-Julia-to-LLVM-Interface)
Julia dynamically links against LLVM by default. Build with `USE_LLVM_SHLIB=0` to link statically.
The code for lowering Julia AST to LLVM IR or interpreting it directly is in directory `src/`.
| File                             | Description                                                          |
|:---------------------------------|:---------------------------------------------------------------------|
| `aotcompile.cpp`                 | Compiler C-interface entry and object file emission                  |
| `builtins.c`                     | Builtin functions                                                    |
| `ccall.cpp`                      | Lowering [`ccall`](../../base/c/#ccall)                              |
| `cgutils.cpp`                    | Lowering utilities, notably for array and tuple accesses             |
| `codegen.cpp`                    | Top-level of code generation, pass list, lowering builtins           |
| `debuginfo.cpp`                  | Tracks debug information for JIT code                                |
| `disasm.cpp`                     | Handles native object file and JIT code diassembly                   |
| `gf.c`                           | Generic functions                                                    |
| `intrinsics.cpp`                 | Lowering intrinsics                                                  |
| `jitlayers.cpp`                  | JIT-specific code, ORC compilation layers/utilities                  |
| `llvm-alloc-helpers.cpp`         | Julia-specific escape analysis                                       |
| `llvm-alloc-opt.cpp`             | Custom LLVM pass to demote heap allocations to the stack             |
| `llvm-cpufeatures.cpp`           | Custom LLVM pass to lower CPU-based functions (e.g. haveFMA)         |
| `llvm-demote-float16.cpp`        | Custom LLVM pass to lower 16b float ops to 32b float ops             |
| `llvm-final-gc-lowering.cpp`     | Custom LLVM pass to lower GC calls to their final form               |
| `llvm-gc-invariant-verifier.cpp` | Custom LLVM pass to verify Julia GC invariants                       |
| `llvm-julia-licm.cpp`            | Custom LLVM pass to hoist/sink Julia-specific intrinsics             |
| `llvm-late-gc-lowering.cpp`      | Custom LLVM pass to root GC-tracked values                           |
| `llvm-lower-handlers.cpp`        | Custom LLVM pass to lower try-catch blocks                           |
| `llvm-muladd.cpp`                | Custom LLVM pass for fast-match FMA                                  |
| `llvm-multiversioning.cpp`       | Custom LLVM pass to generate sysimg code on multiple architectures   |
| `llvm-propagate-addrspaces.cpp`  | Custom LLVM pass to canonicalize addrspaces                          |
| `llvm-ptls.cpp`                  | Custom LLVM pass to lower TLS operations                             |
| `llvm-remove-addrspaces.cpp`     | Custom LLVM pass to remove Julia addrspaces                          |
| `llvm-remove-ni.cpp`             | Custom LLVM pass to remove Julia non-integral addrspaces             |
| `llvm-simdloop.cpp`              | Custom LLVM pass for [`@simd`](../../base/base/#Base.SimdLoop.@simd) |
| `pipeline.cpp`                   | New pass manager pipeline, pass pipeline parsing                     |
| `sys.c`                          | I/O and operating system utility functions                           |


## [Overview of Julia to LLVM Interface](#Overview-of-Julia-to-LLVM-Interface)
Some of the `.cpp` files form a group that compile to a single object.
The difference between an intrinsic and a builtin is that a builtin is a first class function that can be used like any other Julia function. An intrinsic can operate only on unboxed data, and therefore its arguments must be statically typed.


### [Alias Analysis](#LLVM-Alias-Analysis)
Julia currently uses LLVM's [Type Based Alias Analysis](https://llvm.org/docs/LangRef.html#tbaa-metadata). To find the comments that document the inclusion relationships, look for `static MDNode*` in `src/codegen.cpp`.
The `-O` option enables LLVM's [Basic Alias Analysis](https://llvm.org/docs/AliasAnalysis.html#the-basic-aa-pass).


## [Building Julia with a different version of LLVM](#Building-Julia-with-a-different-version-of-LLVM)
The default version of LLVM is specified in `deps/llvm.version`. You can override it by creating a file called `Make.user` in the top-level directory and adding a line to it such as:
```julia
LLVM_VER = 13.0.0
```
Besides the LLVM release numerals, you can also use `DEPS_GIT = llvm` in combination with `USE_BINARYBUILDER_LLVM = 0` to build against the latest development version of LLVM.
You can also specify to build a debug version of LLVM, by setting either `LLVM_DEBUG = 1` or `LLVM_DEBUG = Release` in your `Make.user` file. The former will be a fully unoptimized build of LLVM and the latter will produce an optimized build of LLVM. Depending on your needs the latter will suffice and it quite a bit faster. If you use `LLVM_DEBUG = Release` you will also want to set `LLVM_ASSERTIONS = 1` to enable diagnostics for different passes. Only `LLVM_DEBUG = 1` implies that option by default.


## [Passing options to LLVM](#Passing-options-to-LLVM)
You can pass options to LLVM via the environment variable [`JULIA_LLVM_ARGS`](../../manual/environment-variables/#JULIA_LLVM_ARGS). Here are example settings using `bash` syntax:
-   `export JULIA_LLVM_ARGS=-print-after-all` dumps IR after each pass.
-   `export JULIA_LLVM_ARGS=-debug-only=loop-vectorize` dumps LLVM `DEBUG(...)` diagnostics for loop vectorizer. If you get warnings about "Unknown command line argument", rebuild LLVM with `LLVM_ASSERTIONS = 1`.
-   `export JULIA_LLVM_ARGS=-help` shows a list of available options. `export JULIA_LLVM_ARGS=-help-hidden` shows even more.
-   `export JULIA_LLVM_ARGS="-fatal-warnings -print-options"` is an example how to use multiple options.


### [Useful JULIA_LLVM_ARGS parameters](#Useful-JULIA_LLVM_ARGS-parameters)
-   `-print-after=PASS`: prints the IR after any execution of `PASS`, useful for checking changes done by a pass.
-   `-print-before=PASS`: prints the IR before any execution of `PASS`, useful for checking the input to a pass.
-   `-print-changed`: prints the IR whenever a pass changes the IR, useful for narrowing down which passes are causing problems.
-   `-print-(before|after)=MARKER-PASS`: the Julia pipeline ships with a number of marker passes in the pipeline, which can be used to identify where problems or optimizations are occurring. A marker pass is defined as a pass which appears once in the pipeline and performs no transformations on the IR, and is only useful for targeting print-before/print-after. Currently, the following marker passes exist in the pipeline:
    -   BeforeOptimization
    -   BeforeEarlySimplification
    -   AfterEarlySimplification
    -   BeforeEarlyOptimization
    -   AfterEarlyOptimization
    -   BeforeLoopOptimization
    -   BeforeLICM
    -   AfterLICM
    -   BeforeLoopSimplification
    -   AfterLoopSimplification
    -   AfterLoopOptimization
    -   BeforeScalarOptimization
    -   AfterScalarOptimization
    -   BeforeVectorization
    -   AfterVectorization
    -   BeforeIntrinsicLowering
    -   AfterIntrinsicLowering
    -   BeforeCleanup
    -   AfterCleanup
    -   AfterOptimization
-   `-time-passes`: prints the time spent in each pass, useful for identifying which passes are taking a long time.


### [Useful JULIA_LLVM_ARGS parameters](#Useful-JULIA_LLVM_ARGS-parameters)
-   `-print-module-scope`: used in conjunction with `-print-(before|after)`, gets the entire module rather than the IR unit received by the pass
-   `-debug`: prints out a lot of debugging information throughout LLVM
-   `-debug-only=NAME`, prints out debugging statements from files with `DEBUG_TYPE` defined to `NAME`, useful for getting additional context about a problem


## [Debugging LLVM transformations in isolation](#Debugging-LLVM-transformations-in-isolation)
On occasion, it can be useful to debug LLVM's transformations in isolation from the rest of the Julia system, e.g. because reproducing the issue inside `julia` would take too long, or because one wants to take advantage of LLVM's tooling (e.g. bugpoint).
To start with, you can install the developer tools to work with LLVM via:
```julia
make -C deps install-llvm-tools
```
To get unoptimized IR for the entire system image, pass the `--output-unopt-bc unopt.bc` option to the system image build process, which will output the unoptimized IR to an `unopt.bc` file. This file can then be passed to LLVM tools as usual. `libjulia` can function as an LLVM pass plugin and can be loaded into LLVM tools, to make julia-specific passes available in this environment. In addition, it exposes the `-julia` meta-pass, which runs the entire Julia pass-pipeline over the IR. As an example, to generate a system image with the old pass manager, one could do:
```julia
llc -o sys.o opt.bc
cc -shared -o sys.so sys.o
```
To generate a system image with the new pass manager, one could do:
```julia
opt -load-pass-plugin=libjulia-codegen.so --passes='julia' -o opt.bc unopt.bc
llc -o sys.o opt.bc
cc -shared -o sys.so sys.o
```
This system image can then be loaded by `julia` as usual.
It is also possible to dump an LLVM IR module for just one Julia function, using:


## [Debugging LLVM transformations in isolation](#Debugging-LLVM-transformations-in-isolation)
```julia
fun, T = +, Tuple{Int,Int} # Substitute your function of interest here
optimize = false
open("plus.ll", "w") do file
    println(file, InteractiveUtils._dump_function(fun, T, false, false, false, true, :att, optimize, :default, false))
end
```
These files can be processed the same way as the unoptimized sysimg IR shown above.


## [Running the LLVM test suite](#Running-the-LLVM-test-suite)
To run the llvm tests locally, you need to first install the tools, build julia, then you can run the tests:
```julia
make -C deps install-llvm-tools
make -j julia-src-release
make -C test/llvmpasses
```
If you want to run the individual test files directly, via the commands at the top of each test file, the first step here will have installed the tools into `./usr/tools/opt`. Then you'll want to manually replace `%s` with the name of the test file.


## [Improving LLVM optimizations for Julia](#Improving-LLVM-optimizations-for-Julia)
Improving LLVM code generation usually involves either changing Julia lowering to be more friendly to LLVM's passes, or improving a pass.
If you are planning to improve a pass, be sure to read the [LLVM developer policy](https://llvm.org/docs/DeveloperPolicy.html). The best strategy is to create a code example in a form where you can use LLVM's `opt` tool to study it and the pass of interest in isolation.
1.  Create an example Julia code of interest.
2.  Use `JULIA_LLVM_ARGS=-print-after-all` to dump the IR.
3.  Pick out the IR at the point just before the pass of interest runs.
4.  Strip the debug metadata and fix up the TBAA metadata by hand.
The last step is labor intensive. Suggestions on a better way would be appreciated.


## [The jlcall calling convention](#The-jlcall-calling-convention)
Julia has a generic calling convention for unoptimized code, which looks somewhat as follows:
```c
jl_value_t *any_unoptimized_call(jl_value_t *, jl_value_t **, int);
```
where the first argument is the boxed function object, the second argument is an on-stack array of arguments and the third is the number of arguments. Now, we could perform a straightforward lowering and emit an alloca for the argument array. However, this would betray the SSA nature of the uses at the call site, making optimizations (including GC root placement), significantly harder. Instead, we emit it as follows:
```llvm
call %jl_value_t *@julia.call(jl_value_t *(*)(...) @any_unoptimized_call, %jl_value_t *%arg1, %jl_value_t *%arg2)
```
This allows us to retain the SSA-ness of the uses throughout the optimizer. GC root placement will later lower this call to the original C ABI.


## [GC root placement](#GC-root-placement)
GC root placement is done by an LLVM pass late in the pass pipeline. Doing GC root placement this late enables LLVM to make more aggressive optimizations around code that requires GC roots, as well as allowing us to reduce the number of required GC roots and GC root store operations (since LLVM doesn't understand our GC, it wouldn't otherwise know what it is and is not allowed to do with values stored to the GC frame, so it'll conservatively do very little). As an example, consider an error path
```julia
if some_condition()
    #= Use some variables maybe =#
    error("An error occurred")
end
```
During constant folding, LLVM may discover that the condition is always false, and can remove the basic block. However, if GC root lowering is done early, the GC root slots used in the deleted block, as well as any values kept alive in those slots only because they were used in the error path, would be kept alive by LLVM. By doing GC root lowering late, we give LLVM the license to do any of its usual optimizations (constant folding, dead code elimination, etc.), without having to worry (too much) about which values may or may not be GC tracked.
However, in order to be able to do late GC root placement, we need to be able to identify a) which pointers are GC tracked and b) all uses of such pointers. The goal of the GC placement pass is thus simple:


## [GC root placement](#GC-root-placement)
Minimize the number of needed GC roots/stores to them subject to the constraint that at every safepoint, any live GC-tracked pointer (i.e. for which there is a path after this point that contains a use of this pointer) is in some GC slot.


### [Representation](#Representation)
The primary difficulty is thus choosing an IR representation that allows us to identify GC-tracked pointers and their uses, even after the program has been run through the optimizer. Our design makes use of three LLVM features to achieve this:
-   Custom address spaces
-   Operand Bundles
-   Non-integral pointers
Custom address spaces allow us to tag every point with an integer that needs to be preserved through optimizations. The compiler may not insert casts between address spaces that did not exist in the original program and it must never change the address space of a pointer on a load/store/etc operation. This allows us to annotate which pointers are GC-tracked in an optimizer-resistant way. Note that metadata would not be able to achieve the same purpose. Metadata is supposed to always be discardable without altering the semantics of the program. However, failing to identify a GC-tracked pointer alters the resulting program behavior dramatically - it'll probably crash or return wrong results. We currently use three different address spaces (their numbers are defined in `src/codegen_shared.cpp`):
-   GC Tracked Pointers (currently 10): These are pointers to boxed values that may be put into a GC frame. It is loosely equivalent to a `jl_value_t*` pointer on the C side. N.B. It is illegal to ever have a pointer in this address space that may not be stored to a GC slot.


### [Representation](#Representation)
-   Derived Pointers (currently 11): These are pointers that are derived from some GC tracked pointer. Uses of these pointers generate uses of the original pointer. However, they need not themselves be known to the GC. The GC root placement pass MUST always find the GC tracked pointer from which this pointer is derived and use that as the pointer to root.
-   Callee Rooted Pointers (currently 12): This is a utility address space to express the notion of a callee rooted value. All values of this address space MUST be storable to a GC root (though it is possible to relax this condition in the future), but unlike the other pointers need not be rooted if passed to a call (they do still need to be rooted if they are live across another safepoint between the definition and the call).
-   Pointers loaded from tracked object (currently 13): This is used by arrays, which themselves contain a pointer to the managed data. This data area is owned by the array, but is not a GC-tracked object by itself. The compiler guarantees that as long as this pointer is live, the object that this pointer was loaded from will keep being live.


### [Invariants](#Invariants)
The GC root placement pass makes use of several invariants, which need to be observed by the frontend and are preserved by the optimizer.
First, only the following address space casts are allowed:
-   0->{Tracked,Derived,CalleeRooted}: It is allowable to decay an untracked pointer to any of the others. However, do note that the optimizer has broad license to not root such a value. It is never safe to have a value in address space 0 in any part of the program if it is (or is derived from) a value that requires a GC root.
-   Tracked->Derived: This is the standard decay route for interior values. The placement pass will look for these to identify the base pointer for any use.
-   Tracked->CalleeRooted: Addrspace CalleeRooted serves merely as a hint that a GC root is not required. However, do note that the Derived->CalleeRooted decay is prohibited, since pointers should generally be storable to a GC slot, even in this address space.
Now let us consider what constitutes a use:
-   Loads whose loaded values is in one of the address spaces
-   Stores of a value in one of the address spaces to a location
-   Stores to a pointer in one of the address spaces
-   Calls for which a value in one of the address spaces is an operand
-   Calls in jlcall ABI, for which the argument array contains a value
-   Return instructions.


### [Invariants](#Invariants)
We explicitly allow load/stores and simple calls in address spaces Tracked/Derived. Elements of jlcall argument arrays must always be in address space Tracked (it is required by the ABI that they are valid `jl_value_t*` pointers). The same is true for return instructions (though note that struct return arguments are allowed to have any of the address spaces). The only allowable use of an address space CalleeRooted pointer is to pass it to a call (which must have an appropriately typed operand).
Further, we disallow `getelementptr` in addrspace Tracked. This is because unless the operation is a noop, the resulting pointer will not be validly storable to a GC slot and may thus not be in this address space. If such a pointer is required, it should be decayed to addrspace Derived first.
Lastly, we disallow `inttoptr`/`ptrtoint` instructions in these address spaces. Having these instructions would mean that some `i64` values are really GC tracked. This is problematic, because it breaks that stated requirement that we're able to identify GC-relevant pointers. This invariant is accomplished using the LLVM "non-integral pointers" feature, which is new in LLVM 5.0. It prohibits the optimizer from making optimizations that would introduce these operations. Note we can still insert static constants at JIT time by using `inttoptr` in address space 0 and then decaying to the appropriate address space afterwards.


### [Supporting](#Supporting-%5Bccall%5D(@ref)) [`ccall`](../../base/c/#ccall)
One important aspect missing from the discussion so far is the handling of [`ccall`](../../base/c/#ccall). [`ccall`](../../base/c/#ccall) has the peculiar feature that the location and scope of a use do not coincide. As an example consider:
```julia
A = randn(1024)
ccall(:foo, Cvoid, (Ptr{Float64},), A)
```
In lowering, the compiler will insert a conversion from the array to the pointer which drops the reference to the array value. However, we of course need to make sure that the array does stay alive while we're doing the [`ccall`](../../base/c/#ccall). To understand how this is done, lets look at a hypothetical approximate possible lowering of the above code:
```julia
return $(Expr(:foreigncall, :(:foo), Cvoid, svec(Ptr{Float64}), 0, :(:ccall), Expr(:foreigncall, :(:jl_array_ptr), Ptr{Float64}, svec(Any), 0, :(:ccall), :(A)), :(A)))
```
The last `:(A)`, is an extra argument list inserted during lowering that informs the code generator which Julia level values need to be kept alive for the duration of this [`ccall`](../../base/c/#ccall). We then take this information and represent it in an "operand bundle" at the IR level. An operand bundle is essentially a fake use that is attached to the call site. At the IR level, this looks like so:
```llvm
call void inttoptr (i64 ... to void (double*)*)(double* %5) [ "jl_roots"(%jl_value_t addrspace(10)* %A) ]
```


### [Supporting](#Supporting-%5Bccall%5D(@ref)) [`ccall`](../../base/c/#ccall)
The GC root placement pass will treat the `jl_roots` operand bundle as if it were a regular operand. However, as a final step, after the GC roots are inserted, it will drop the operand bundle to avoid confusing instruction selection.


### [Supporting](#Supporting-%5Bpointer_from_objref%5D(@ref)) [`pointer_from_objref`](../../base/c/#Base.pointer_from_objref)
[`pointer_from_objref`](../../base/c/#Base.pointer_from_objref) is special because it requires the user to take explicit control of GC rooting. By our above invariants, this function is illegal, because it performs an address space cast from 10 to 0. However, it can be useful, in certain situations, so we provide a special intrinsic:
```llvm
declared %jl_value_t *julia.pointer_from_objref(%jl_value_t addrspace(10)*)
```
which is lowered to the corresponding address space cast after GC root lowering. Do note however that by using this intrinsic, the caller assumes all responsibility for making sure that the value in question is rooted. Further this intrinsic is not considered a use, so the GC root placement pass will not provide a GC root for the function. As a result, the external rooting must be arranged while the value is still tracked by the system. I.e. it is not valid to attempt to use the result of this operation to establish a global root - the optimizer may have already dropped the value.


### [Keeping values alive in the absence of uses](#Keeping-values-alive-in-the-absence-of-uses)
In certain cases it is necessary to keep an object alive, even though there is no compiler-visible use of said object. This may be case for low level code that operates on the memory-representation of an object directly or code that needs to interface with C code. In order to allow this, we provide the following intrinsics at the LLVM level:
```julia
token @llvm.julia.gc_preserve_begin(...)
void @llvm.julia.gc_preserve_end(token)
```
(The `llvm.` in the name is required in order to be able to use the `token` type). The semantics of these intrinsics are as follows: At any safepoint that is dominated by a `gc_preserve_begin` call, but that is not not dominated by a corresponding `gc_preserve_end` call (i.e. a call whose argument is the token returned by a `gc_preserve_begin` call), the values passed as arguments to that `gc_preserve_begin` will be kept live. Note that the `gc_preserve_begin` still counts as a regular use of those values, so the standard lifetime semantics will ensure that the values will be kept alive before entering the preserve region.
------------------------------------------------------------------------


# printf() and stdio in the Julia runtime · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/stdio/


# [printf() and stdio in the Julia runtime](#printf()-and-stdio-in-the-Julia-runtime)


## [Libuv wrappers for stdio](#Libuv-wrappers-for-stdio)
`julia.h` defines [libuv](https://docs.libuv.org) wrappers for the `stdio.h` streams:
```c
uv_stream_t *JL_STDIN;
uv_stream_t *JL_STDOUT;
uv_stream_t *JL_STDERR;
```
... and corresponding output functions:
```c
int jl_printf(uv_stream_t *s, const char *format, ...);
int jl_vprintf(uv_stream_t *s, const char *format, va_list args);
```
These `printf` functions are used by the `.c` files in the `src/` and `cli/` directories wherever stdio is needed to ensure that output buffering is handled in a unified way.
In special cases, like signal handlers, where the full libuv infrastructure is too heavy, `jl_safe_printf()` can be used to [`write(2)`](../../base/file/#Base.write-Tuple%7BString,%20Any%7D) directly to `STDERR_FILENO`:
```c
void jl_safe_printf(const char *str, ...);
```


## [Interface between JL_STD* and Julia code](#Interface-between-JL_STD*-and-Julia-code)
[`Base.stdin`](../../base/io-network/#Base.stdin), [`Base.stdout`](../../base/io-network/#Base.stdout) and [`Base.stderr`](../../base/io-network/#Base.stderr) are bound to the `JL_STD*` libuv streams defined in the runtime.
Julia's `__init__()` function (in `base/sysimg.jl`) calls `reinit_stdio()` (in `base/stream.jl`) to create Julia objects for [`Base.stdin`](../../base/io-network/#Base.stdin), [`Base.stdout`](../../base/io-network/#Base.stdout) and [`Base.stderr`](../../base/io-network/#Base.stderr).
`reinit_stdio()` uses [`ccall`](../../base/c/#ccall) to retrieve pointers to `JL_STD*` and calls `jl_uv_handle_type()` to inspect the type of each stream. It then creates a Julia `Base.IOStream`, `Base.TTY` or `Base.PipeEndpoint` object to represent each stream, e.g.:
```julia
$ julia -e 'println(typeof((stdin, stdout, stderr)))'
Tuple{Base.TTY,Base.TTY,Base.TTY}

$ julia -e 'println(typeof((stdin, stdout, stderr)))' < /dev/null 2>/dev/null
Tuple{IOStream,Base.TTY,IOStream}

$ echo hello | julia -e 'println(typeof((stdin, stdout, stderr)))' | cat
Tuple{Base.PipeEndpoint,Base.PipeEndpoint,Base.TTY}
```
The [`Base.read`](../../base/io-network/#Base.read) and [`Base.write`](../../base/io-network/#Base.write) methods for these streams use [`ccall`](../../base/c/#ccall) to call libuv wrappers in `src/jl_uv.c`, e.g.:


## [Interface between JL_STD* and Julia code](#Interface-between-JL_STD*-and-Julia-code)
```julia
stream.jl: function write(s::IO, p::Ptr, nb::Integer)
               -> ccall(:jl_uv_write, ...)
  jl_uv.c:          -> int jl_uv_write(uv_stream_t *stream, ...)
                        -> uv_write(uvw, stream, buf, ...)
```


## [printf() during initialization](#printf()-during-initialization)
The libuv streams relied upon by `jl_printf()` etc., are not available until midway through initialization of the runtime (see `init.c`, `init_stdio()`). Error messages or warnings that need to be printed before this are routed to the standard C library `fwrite()` function by the following mechanism:
In `sys.c`, the `JL_STD*` stream pointers are statically initialized to integer constants: `STD*_FILENO (0, 1 and 2)`. In `jl_uv.c` the `jl_uv_puts()` function checks its `uv_stream_t* stream` argument and calls `fwrite()` if stream is set to `STDOUT_FILENO` or `STDERR_FILENO`.
This allows for uniform use of `jl_printf()` throughout the runtime regardless of whether or not any particular piece of code is reachable before initialization is complete.


## [Legacy ios.c library](#Legacy-ios.c-library)
The `src/support/ios.c` library is inherited from [femtolisp](https://github.com/JeffBezanson/femtolisp). It provides cross-platform buffered file IO and in-memory temporary buffers.
`ios.c` is still used by:
-   `src/flisp/*.c`
-   `src/dump.c` – for serialization file IO and for memory buffers.
-   `src/staticdata.c` – for serialization file IO and for memory buffers.
-   `base/iostream.jl` – for file IO (see `base/fs.jl` for libuv equivalent).
Use of `ios.c` in these modules is mostly self-contained and separated from the libuv I/O system. However, there is [one place](https://github.com/JuliaLang/julia/blob/master/src/flisp/print.c#L654) where femtolisp calls through to `jl_printf()` with a legacy `ios_t` stream.
There is a hack in `ios.h` that makes the `ios_t.bm` field line up with the `uv_stream_t.type` and ensures that the values used for `ios_t.bm` to not overlap with valid `UV_HANDLE_TYPE` values. This allows `uv_stream_t` pointers to point to `ios_t` streams.
This is needed because `jl_printf()` caller `jl_static_show()` is passed an `ios_t` stream by femtolisp's `fl_print()` function. Julia's `jl_uv_puts()` function has special handling for this:
```c
if (stream->type > UV_HANDLE_TYPE_MAX) {
    return ios_write((ios_t*)stream, str, n);
}
```
------------------------------------------------------------------------


# Bounds checking · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/boundscheck/


# [Bounds checking](#Bounds-checking)
Like many modern programming languages, Julia uses bounds checking to ensure program safety when accessing arrays. In tight inner loops or other performance critical situations, you may wish to skip these bounds checks to improve runtime performance. For instance, in order to emit vectorized (SIMD) instructions, your loop body cannot contain branches, and thus cannot contain bounds checks. Consequently, Julia includes an `@inbounds(...)` macro to tell the compiler to skip such bounds checks within the given block. User-defined array types can use the `@boundscheck(...)` macro to achieve context-sensitive code selection.


## [Eliding bounds checks](#Eliding-bounds-checks)
The `@boundscheck(...)` macro marks blocks of code that perform bounds checking. When such blocks are inlined into an `@inbounds(...)` block, the compiler may remove these blocks. The compiler removes the `@boundscheck` block *only if it is inlined* into the calling function. For example, you might write the method `sum` as:
```julia
function sum(A::AbstractArray)
    r = zero(eltype(A))
    for i in eachindex(A)
        @inbounds r += A[i]
    end
    return r
end
```
With a custom array-like type `MyArray` having:
```julia
@inline getindex(A::MyArray, i::Real) = (@boundscheck checkbounds(A, i); A.data[to_index(i)])
```
Then when `getindex` is inlined into `sum`, the call to `checkbounds(A, i)` will be elided. If your function contains multiple layers of inlining, only `@boundscheck` blocks at most one level of inlining deeper are eliminated. The rule prevents unintended changes in program behavior from code further up the stack.


### [Caution!](#Caution!)
It is easy to accidentally expose unsafe operations with `@inbounds`. You might be tempted to write the above example as
```julia
function sum(A::AbstractArray)
    r = zero(eltype(A))
    for i in 1:length(A)
        @inbounds r += A[i]
    end
    return r
end
```
Which quietly assumes 1-based indexing and therefore exposes unsafe memory access when used with [`OffsetArrays`](../offset-arrays/#man-custom-indices):
```julia-repl
julia> using OffsetArrays

julia> sum(OffsetArray([1, 2, 3], -10))
9164911648 # inconsistent results or segfault
```
While the original source of the error here is `1:length(A)`, the use of `@inbounds` increases the consequences from a bounds error to a less easily caught and debugged unsafe memory access. It is often difficult or impossible to prove that a method which uses `@inbounds` is safe, so one must weigh the benefits of performance improvements against the risk of segfaults and silent misbehavior, especially in public facing APIs.


## [Propagating inbounds](#Propagating-inbounds)
There may be certain scenarios where for code-organization reasons you want more than one layer between the `@inbounds` and `@boundscheck` declarations. For instance, the default `getindex` methods have the chain `getindex(A::AbstractArray, i::Real)` calls `getindex(IndexStyle(A), A, i)` calls `_getindex(::IndexLinear, A, i)`.
To override the "one layer of inlining" rule, a function may be marked with [`Base.@propagate_inbounds`](../../base/base/#Base.@propagate_inbounds) to propagate an inbounds context (or out of bounds context) through one additional layer of inlining.


## [The bounds checking call hierarchy](#The-bounds-checking-call-hierarchy)
The overall hierarchy is:
-   `checkbounds(A, I...)` which calls
    -   `checkbounds(Bool, A, I...)` which calls
        -   `checkbounds_indices(Bool, axes(A), I)` which recursively calls
            -   `checkindex` for each dimension
Here `A` is the array, and `I` contains the "requested" indices. `axes(A)` returns a tuple of "permitted" indices of `A`.
`checkbounds(A, I...)` throws an error if the indices are invalid, whereas `checkbounds(Bool, A, I...)` returns `false` in that circumstance. `checkbounds_indices` discards any information about the array other than its `axes` tuple, and performs a pure indices-vs-indices comparison: this allows relatively few compiled methods to serve a huge variety of array types. Indices are specified as tuples, and are usually compared in a 1-1 fashion with individual dimensions handled by calling another important function, `checkindex`: typically,
```julia
checkbounds_indices(Bool, (IA1, IA...), (I1, I...)) = checkindex(Bool, IA1, I1) &
                                                      checkbounds_indices(Bool, IA, I)
```
so `checkindex` checks a single dimension. All of these functions, including the unexported `checkbounds_indices` have docstrings accessible with `?` .
If you have to customize bounds checking for a specific array type, you should specialize `checkbounds(Bool, A, I...)`. However, in most cases you should be able to rely on `checkbounds_indices` as long as you supply useful `axes` for your array type.


## [The bounds checking call hierarchy](#The-bounds-checking-call-hierarchy)
If you have novel index types, first consider specializing `checkindex`, which handles a single index for a particular dimension of an array. If you have a custom multidimensional index type (similar to `CartesianIndex`), then you may have to consider specializing `checkbounds_indices`.
Note this hierarchy has been designed to reduce the likelihood of method ambiguities. We try to make `checkbounds` the place to specialize on array type, and try to avoid specializations on index types; conversely, `checkindex` is intended to be specialized only on index type (especially, the last argument).


## [Emit bounds checks](#Emit-bounds-checks)
Julia can be launched with `--check-bounds={yes|no|auto}` to emit bounds checks always, never, or respect `@inbounds` declarations.
------------------------------------------------------------------------


# Proper maintenance and care of multi-threading locks · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/locks/


# [Proper maintenance and care of multi-threading locks](#Proper-maintenance-and-care-of-multi-threading-locks)
The following strategies are used to ensure that the code is dead-lock free (generally by addressing the 4th Coffman condition: circular wait).
> 1.  structure code such that only one lock will need to be acquired at a time
> 2.  always acquire shared locks in the same order, as given by the table below
> 3.  avoid constructs that expect to need unrestricted recursion


## [Locks](#Locks)
Below are all of the locks that exist in the system and the mechanisms for using them that avoid the potential for deadlocks (no Ostrich algorithm allowed here):
The following are definitely leaf locks (level 1), and must not try to acquire any other lock:
> -   safepoint
>
>     > Note that this lock is acquired implicitly by `JL_LOCK` and `JL_UNLOCK`. use the `_NOGC` variants to avoid that for level 1 locks.
>     >
>     > While holding this lock, the code must not do any allocation or hit any safepoints. Note that there are safepoints when doing allocation, enabling / disabling GC, entering / restoring exception frames, and taking / releasing locks.
>
> -   shared_map
>
> -   finalizers
>
> -   pagealloc
>
> -   gc*perm*lock
>
> -   flisp
>
> -   jl*in*stackwalk (Win32)
>
> -   ResourcePool\<?>::mutex
>
> -   RLST_mutex
>
> -   llvm*printing*mutex
>
> -   jl*locked*stream::mutex
>
> -   debuginfo_asyncsafe
>
> -   inference*timing*mutex
>
> -   ExecutionEngine::SessionLock
>
>     > flisp itself is already threadsafe, this lock only protects the `jl_ast_context_list_t` pool likewise, the ResourcePool\<?>::mutexes just protect the associated resource pool
The following is a leaf lock (level 2), and only acquires level 1 locks (safepoint) internally:
> -   global*roots*lock
> -   Module->lock
> -   JLDebuginfoPlugin::PluginMutex
> -   newly*inferred*mutex
The following is a level 3 lock, which can only acquire level 1 or level 2 locks internally:
> -   Method->writelock
> -   typecache


## [Locks](#Locks)
The following is a level 4 lock, which can only recurse to acquire level 1, 2, or 3 locks:
> -   MethodTable->writelock
No Julia code may be called while holding a lock above this point.
orc::ThreadSafeContext (TSCtx) locks occupy a special spot in the locking hierarchy. They are used to protect LLVM's global non-threadsafe state, but there may be an arbitrary number of them. By default, all of these locks may be treated as level 5 locks for the purposes of comparing with the rest of the hierarchy. Acquiring a TSCtx should only be done from the JIT's pool of TSCtx's, and all locks on that TSCtx should be released prior to returning it to the pool. If multiple TSCtx locks must be acquired at the same time (due to recursive compilation), then locks should be acquired in the order that the TSCtxs were borrowed from the pool.
The following is a level 5 lock
> -   JuliaOJIT::EmissionMutex
The following are a level 6 lock, which can only recurse to acquire locks at lower levels:
> -   codegen
> -   jl*modules*mutex
The following is an almost root lock (level end-1), meaning only the root look may be held when trying to acquire it:
> -   typeinf
>
>     > this one is perhaps one of the most tricky ones, since type-inference can be invoked from many points
>     >
>     > currently the lock is merged with the codegen lock, since they call each other recursively


## [Locks](#Locks)
The following lock synchronizes IO operation. Be aware that doing any I/O (for example, printing warning messages or debug information) while holding any other lock listed above may result in pernicious and hard-to-find deadlocks. BE VERY CAREFUL!
> -   iolock
>
> -   Individual ThreadSynchronizers locks
>
>     > this may continue to be held after releasing the iolock, or acquired without it, but be very careful to never attempt to acquire the iolock while holding it
>
> -   Libdl.LazyLibrary lock
The following is the root lock, meaning no other lock shall be held when trying to acquire it:
> -   toplevel
>
>     > this should be held while attempting a top-level action (such as making a new type or defining a new method): trying to obtain this lock inside a staged function will cause a deadlock condition!
>     >
>     > additionally, it's unclear if *any* code can safely run in parallel with an arbitrary toplevel expression, so it may require all threads to get to a safepoint first


## [Broken Locks](#Broken-Locks)
The following locks are broken:
-   toplevel
    > doesn't exist right now
    >
    > fix: create it
-   Module->lock
    > This is vulnerable to deadlocks since it can't be certain it is acquired in sequence. Some operations (such as `import_module`) are missing a lock.
    >
    > fix: replace with `jl_modules_mutex`?
-   loading.jl: `require` and `register_root_module`
    > This file potentially has numerous problems.
    >
    > fix: needs locks


## [Shared Global Data Structures](#Shared-Global-Data-Structures)
These data structures each need locks due to being shared mutable global state. It is the inverse list for the above lock priority list. This list does not include level 1 leaf resources due to their simplicity.
MethodTable modifications (def, cache) : MethodTable->writelock
Type declarations : toplevel lock
Type application : typecache lock
Global variable tables : Module->lock
Module serializer : toplevel lock
JIT & type-inference : codegen lock
MethodInstance/CodeInstance updates : Method->writelock, codegen lock
> -   These are set at construction and immutable:
>     -   specTypes
>     -   sparam_vals
>     -   def
>     -   owner
> -   These are set by `jl_type_infer` (while holding codegen lock):
>     -   cache
>     -   rettype
>     -   inferred
```julia
    * valid ages
```
> -   `inInference` flag:
>     -   optimization to quickly avoid recurring into `jl_type_infer` while it is already running
>     -   actual state (of setting `inferred`, then `fptr`) is protected by codegen lock


## [Shared Global Data Structures](#Shared-Global-Data-Structures)
> -   Function pointers:
>
>     -   these transition once, from `NULL` to a value, while the codegen lock is held
>
> -   Code-generator cache (the contents of `functionObjectsDecls`):
>
>     -   these can transition multiple times, but only while the codegen lock is held
>     -   it is valid to use old version of this, or block for new versions of this, so races are benign, as long as the code is careful not to reference other data in the method instance (such as `rettype`) and assume it is coordinated, unless also holding the codegen lock
LLVMContext : codegen lock
Method : Method->writelock
-   roots array (serializer and codegen)
-   invoke / specializations / tfunc modifications
------------------------------------------------------------------------


# Arrays with custom indices · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/offset-arrays/


# [Arrays with custom indices](#man-custom-indices)
Conventionally, Julia's arrays are indexed starting at 1, whereas some other languages start numbering at 0, and yet others (e.g., Fortran) allow you to specify arbitrary starting indices. While there is much merit in picking a standard (i.e., 1 for Julia), there are some algorithms which simplify considerably if you can index outside the range `1:size(A,d)` (and not just `0:size(A,d)-1`, either). To facilitate such computations, Julia supports arrays with arbitrary indices.
The purpose of this page is to address the question, "what do I have to do to support such arrays in my own code?" First, let's address the simplest case: if you know that your code will never need to handle arrays with unconventional indexing, hopefully the answer is "nothing." Old code, on conventional arrays, should function essentially without alteration as long as it was using the exported interfaces of Julia. If you find it more convenient to just force your users to supply traditional arrays where indexing starts at one, you can add
```julia
Base.require_one_based_indexing(arrays...)
```
where `arrays...` is a list of the array objects that you wish to check for anything that violates 1-based indexing.


## [Generalizing existing code](#Generalizing-existing-code)
As an overview, the steps are:
-   replace many uses of `size` with `axes`
-   replace `1:length(A)` with `eachindex(A)`, or in some cases `LinearIndices(A)`
-   replace explicit allocations like `Array{Int}(undef, size(B))` with `similar(Array{Int}, axes(B))`
These are described in more detail below.


### [Things to watch out for](#Things-to-watch-out-for)
Because unconventional indexing breaks many people's assumptions that all arrays start indexing with 1, there is always the chance that using such arrays will trigger errors. The most frustrating bugs would be incorrect results or segfaults (total crashes of Julia). For example, consider the following function:
```julia
function mycopy!(dest::AbstractVector, src::AbstractVector)
    length(dest) == length(src) || throw(DimensionMismatch("vectors must match"))
    # OK, now we're safe to use @inbounds, right? (not anymore!)
    for i = 1:length(src)
        @inbounds dest[i] = src[i]
    end
    dest
end
```
This code implicitly assumes that vectors are indexed from 1; if `dest` starts at a different index than `src`, there is a chance that this code would trigger a segfault. (If you do get segfaults, to help locate the cause try running julia with the option `--check-bounds=yes`.)


### [Using axes for bounds checks and loop iteration](#Using-axes-for-bounds-checks-and-loop-iteration)
`axes(A)` (reminiscent of `size(A)`) returns a tuple of `AbstractUnitRange{<:Integer}` objects, specifying the range of valid indices along each dimension of `A`. When `A` has unconventional indexing, the ranges may not start at 1. If you just want the range for a particular dimension `d`, there is `axes(A, d)`.
Base implements a custom range type, `OneTo`, where `OneTo(n)` means the same thing as `1:n` but in a form that guarantees (via the type system) that the lower index is 1. For any new [`AbstractArray`](../../base/arrays/#Core.AbstractArray) type, this is the default returned by `axes`, and it indicates that this array type uses "conventional" 1-based indexing.
For bounds checking, note that there are dedicated functions `checkbounds` and `checkindex` which can sometimes simplify such tests.


### [Linear indexing (LinearIndices)](#Linear-indexing-(LinearIndices))
Some algorithms are most conveniently (or efficiently) written in terms of a single linear index, `A[i]` even if `A` is multi-dimensional. Regardless of the array's native indices, linear indices always range from `1:length(A)`. However, this raises an ambiguity for one-dimensional arrays (a.k.a., [`AbstractVector`](../../base/arrays/#Base.AbstractVector)): does `v[i]` mean linear indexing , or Cartesian indexing with the array's native indices?
For this reason, your best option may be to iterate over the array with `eachindex(A)`, or, if you require the indices to be sequential integers, to get the index range by calling `LinearIndices(A)`. This will return `axes(A, 1)` if A is an AbstractVector, and the equivalent of `1:length(A)` otherwise.
By this definition, 1-dimensional arrays always use Cartesian indexing with the array's native indices. To help enforce this, it's worth noting that the index conversion functions will throw an error if shape indicates a 1-dimensional array with unconventional indexing (i.e., is a `Tuple{UnitRange}` rather than a tuple of `OneTo`). For arrays with conventional indexing, these functions continue to work the same as always.
Using `axes` and `LinearIndices`, here is one way you could rewrite `mycopy!`:
```julia
function mycopy!(dest::AbstractVector, src::AbstractVector)
    axes(dest) == axes(src) || throw(DimensionMismatch("vectors must match"))
    for i in LinearIndices(src)
        @inbounds dest[i] = src[i]
    end
    dest
end
```


### [Allocating storage using generalizations of similar](#Allocating-storage-using-generalizations-of-similar)
Storage is often allocated with `Array{Int}(undef, dims)` or `similar(A, args...)`. When the result needs to match the indices of some other array, this may not always suffice. The generic replacement for such patterns is to use `similar(storagetype, shape)`. `storagetype` indicates the kind of underlying "conventional" behavior you'd like, e.g., `Array{Int}` or `BitArray` or even `dims->zeros(Float32, dims)` (which would allocate an all-zeros array). `shape` is a tuple of [`Integer`](../../base/numbers/#Core.Integer) or `AbstractUnitRange` values, specifying the indices that you want the result to use. Note that a convenient way of producing an all-zeros array that matches the indices of A is simply `zeros(A)`.
Let's walk through a couple of explicit examples. First, if `A` has conventional indices, then `similar(Array{Int}, axes(A))` would end up calling `Array{Int}(undef, size(A))`, and thus return an array. If `A` is an `AbstractArray` type with unconventional indexing, then `similar(Array{Int}, axes(A))` should return something that "behaves like" an `Array{Int}` but with a shape (including indices) that matches `A`. (The most obvious implementation is to allocate an `Array{Int}(undef, size(A))` and then "wrap" it in a type that shifts the indices.)
Note also that `similar(Array{Int}, (axes(A, 2),))` would allocate an `AbstractVector{Int}` (i.e., 1-dimensional array) that matches the indices of the columns of `A`.


## [Writing custom array types with non-1 indexing](#Writing-custom-array-types-with-non-1-indexing)
Most of the methods you'll need to define are standard for any `AbstractArray` type, see [Abstract Arrays](../../manual/interfaces/#man-interface-array). This page focuses on the steps needed to define unconventional indexing.


### [Custom AbstractUnitRange types](#Custom-AbstractUnitRange-types)
If you're writing a non-1 indexed array type, you will want to specialize `axes` so it returns a `UnitRange`, or (perhaps better) a custom `AbstractUnitRange`. The advantage of a custom type is that it "signals" the allocation type for functions like `similar`. If we're writing an array type for which indexing will start at 0, we likely want to begin by creating a new `AbstractUnitRange`, `ZeroRange`, where `ZeroRange(n)` is equivalent to `0:n-1`.
In general, you should probably *not* export `ZeroRange` from your package: there may be other packages that implement their own `ZeroRange`, and having multiple distinct `ZeroRange` types is (perhaps counterintuitively) an advantage: `ModuleA.ZeroRange` indicates that `similar` should create a `ModuleA.ZeroArray`, whereas `ModuleB.ZeroRange` indicates a `ModuleB.ZeroArray` type. This design allows peaceful coexistence among many different custom array types.
Note that the Julia package [CustomUnitRanges.jl](https://github.com/JuliaArrays/CustomUnitRanges.jl) can sometimes be used to avoid the need to write your own `ZeroRange` type.


### [Specializing axes](#Specializing-axes)
Once you have your `AbstractUnitRange` type, then specialize `axes`:
```julia
Base.axes(A::ZeroArray) = map(n->ZeroRange(n), A.size)
```
where here we imagine that `ZeroArray` has a field called `size` (there would be other ways to implement this).
In some cases, the fallback definition for `axes(A, d)`:
```julia
axes(A::AbstractArray{T,N}, d) where {T,N} = d <= N ? axes(A)[d] : OneTo(1)
```
may not be what you want: you may need to specialize it to return something other than `OneTo(1)` when `d > ndims(A)`. Likewise, in `Base` there is a dedicated function `axes1` which is equivalent to `axes(A, 1)` but which avoids checking (at runtime) whether `ndims(A) > 0`. (This is purely a performance optimization.) It is defined as:
```julia
axes1(A::AbstractArray{T,0}) where {T} = OneTo(1)
axes1(A::AbstractArray) = axes(A)[1]
```
If the first of these (the zero-dimensional case) is problematic for your custom array type, be sure to specialize it appropriately.