## [Overview](#Overview)
![Diagram of the compiler flow](../img/compiler_diagram.png)
Codegen produces an LLVM module containing IR for one or more Julia functions from the original Julia SSA IR produced by type inference (labeled as translate on the compiler diagram above). It also produces a mapping of code-instance to LLVM function name. However, though some optimizations have been applied by the Julia-based compiler on Julia IR, the LLVM IR produced by codegen still contains many opportunities for optimization. Thus, the first step the JIT takes is to run a target-independent optimization pipeline<sup>[[tdp\]](#footnote-tdp)</sup> on the LLVM module. Then, the JIT runs a target-dependent optimization pipeline, which includes target-specific optimizations and code generation, and outputs an object file. Finally, the JIT links the resulting object file into the current process and makes the code available for execution. All of this is controlled by code in `src/jitlayers.cpp`.
Currently, only one thread at a time is permitted to enter the optimize-compile-link pipeline at a time, due to restrictions imposed by one of our linkers (RuntimeDyld). However, the JIT is designed to support concurrent optimization and compilation, and the linker restriction is expected to be lifted in the future when RuntimeDyld has been fully superseded on all platforms.


## [Optimization Pipeline](#Optimization-Pipeline)
The optimization pipeline is based off LLVM's new pass manager, but the pipeline is customized for Julia's needs. The pipeline is defined in `src/pipeline.cpp`, and broadly proceeds through a number of stages as detailed below.
1.  Early Simplification
    1.  These passes are mainly used to simplify the IR and canonicalize patterns so that later passes can identify those patterns more easily. Additionally, various intrinsic calls such as branch prediction hints and annotations are lowered into other metadata or other IR features. [`SimplifyCFG`](https://llvm.org/docs/Passes.html#simplifycfg-simplify-the-cfg) (simplify control flow graph), [`DCE`](https://llvm.org/docs/Passes.html#dce-dead-code-elimination) (dead code elimination), and [`SROA`](https://llvm.org/docs/Passes.html#sroa-scalar-replacement-of-aggregates) (scalar replacement of aggregates) are some of the key players here.
2.  Early Optimization
    1.  These passes are typically cheap and are primarily focused around reducing the number of instructions in the IR and propagating knowledge to other instructions. For example, [`EarlyCSE`](https://en.wikipedia.org/wiki/Common_subexpression_elimination) is used to perform common subexpression elimination, and [`InstCombine`](https://llvm.org/docs/Passes.html#instcombine-combine-redundant-instructions) and [`InstSimplify`](https://llvm.org/doxygen/classllvm_1_1InstSimplifyPass.html#details) perform a number of small peephole optimizations to make operations less expensive.
3.  Loop Optimization


## [Optimization Pipeline](#Optimization-Pipeline)
    1.  These passes canonicalize and simplify loops. Loops are often hot code, which makes loop optimization extremely important for performance. Key players here include [`LoopRotate`](https://llvm.org/docs/Passes.html#loop-rotate-rotate-loops), [`LICM`](https://llvm.org/docs/Passes.html#licm-loop-invariant-code-motion), and [`LoopFullUnroll`](https://llvm.org/docs/Passes.html#loop-unroll-unroll-loops). Some bounds check elimination also happens here, as a result of the [`IRCE`](https://llvm.org/doxygen/InductiveRangeCheckElimination_8cpp_source.html) pass which can prove certain bounds are never exceeded.
4.  Scalar Optimization
    1.  The scalar optimization pipeline contains a number of more expensive, but more powerful passes such as [`GVN`](https://llvm.org/docs/Passes.html#gvn-global-value-numbering) (global value numbering), [`SCCP`](https://llvm.org/docs/Passes.html#sccp-sparse-conditional-constant-propagation) (sparse conditional constant propagation), and another round of bounds check elimination. These passes are expensive, but they can often remove large amounts of code and make vectorization much more successful and effective. Several other simplification and optimization passes intersperse the more expensive ones to reduce the amount of work they have to do.
5.  Vectorization


## [Optimization Pipeline](#Optimization-Pipeline)
    1.  [Automatic vectorization](https://en.wikipedia.org/wiki/Automatic_vectorization) is an extremely powerful transformation for CPU-intensive code. Briefly, vectorization allows execution of a [single instruction on multiple data](https://en.wikipedia.org/wiki/Single_instruction,_multiple_data) (SIMD), e.g. performing 8 addition operations at the same time. However, proving code to be both capable of vectorization and profitable to vectorize is difficult, and this relies heavily on the prior optimization passes to massage the IR into a state where vectorization is worth it.
6.  Intrinsic Lowering
    1.  Julia inserts a number of custom intrinsics, for reasons such as object allocation, garbage collection, and exception handling. These intrinsics were originally placed to make optimization opportunities more obvious, but they are now lowered into LLVM IR to enable the IR to be emitted as machine code.
7.  Cleanup
    1.  These passes are last-chance optimizations, and perform small optimizations such as fused multiply-add propagation and division-remainder simplification. Additionally, targets that do not support half-precision floating point numbers will have their half-precision instructions lowered into single-precision instructions here, and passes are added to provide sanitizer support.


## [Target-Dependent Optimization and Code Generation](#Target-Dependent-Optimization-and-Code-Generation)
LLVM provides target-dependent optimization and machine code generation in the same pipeline, located in the TargetMachine for a given platform. These passes include instruction selection, instruction scheduling, register allocation, and machine code emission. The LLVM documentation provides a good overview of the process, and the LLVM source code is the best place to look for details on the pipeline and passes.


## [Linking](#Linking)
Currently, Julia is transitioning between two linkers: the older RuntimeDyld linker, and the newer [JITLink](https://llvm.org/docs/JITLink.html) linker. JITLink contains a number of features that RuntimeDyld does not have, such as concurrent and reentrant linking, but currently lacks good support for profiling integrations and does not yet support all of the platforms that RuntimeDyld supports. Over time, JITLink is expected to replace RuntimeDyld entirely. Further details on JITLink can be found in the LLVM documentation.


## [Execution](#Execution)
Once the code has been linked into the current process, it is available for execution. This fact is made known to the generating codeinst by updating the `invoke`, `specsigflags`, and `specptr` fields appropriately. Codeinsts support upgrading `invoke`, `specsigflags`, and `specptr` fields, so long as every combination of these fields that exists at any given point in time is valid to be called. This allows the JIT to update these fields without invalidating existing codeinsts, supporting a potential future concurrent JIT. Specifically, the following states may be valid:
1.  `invoke` is NULL, `specsigflags` is 0b00, `specptr` is NULL
    1.  This is the initial state of a codeinst, and indicates that the codeinst has not yet been compiled.
2.  `invoke` is non-null, `specsigflags` is 0b00, `specptr` is NULL
    1.  This indicates that the codeinst was not compiled with any specialization, and that the codeinst should be invoked directly. Note that in this instance, `invoke` does not read either the `specsigflags` or `specptr` fields, and therefore they may be modified without invalidating the `invoke` pointer.
3.  `invoke` is non-null, `specsigflags` is 0b10, `specptr` is non-null
    1.  This indicates that the codeinst was compiled, but a specialized function signature was deemed unnecessary by codegen.
4.  `invoke` is non-null, `specsigflags` is 0b11, `specptr` is non-null


## [Execution](#Execution)
    1.  This indicates that the codeinst was compiled, and a specialized function signature was deemed necessary by codegen. The `specptr` field contains a pointer to the specialized function signature. The `invoke` pointer is permitted to read both `specsigflags` and `specptr` fields.
In addition, there are a number of different transitional states that occur during the update process. To account for these potential situations, the following write and read patterns should be used when dealing with these codeinst fields.
1.  When writing `invoke`, `specsigflags`, and `specptr`:
    1.  Perform an atomic compare-exchange operation of specptr assuming the old value was NULL. This compare-exchange operation should have at least acquire-release ordering, to provide ordering guarantees of the remaining memory operations in the write.
    2.  If `specptr` was non-null, cease the write operation and wait for bit 0b10 of `specsigflags` to be written.
    3.  Write the new low bit of `specsigflags` to its final value. This may be a relaxed write.
    4.  Write the new `invoke` pointer to its final value. This must have at least a release memory ordering to synchronize with reads of `invoke`.
    5.  Set the second bit of `specsigflags` to 1. This must be at least a release memory ordering to synchronize with reads of `specsigflags`. This step completes the write operation and announces to all other threads that all fields have been set.
2.  When reading all of `invoke`, `specsigflags`, and `specptr`:


## [Execution](#Execution)
    1.  Read the `invoke` field with at least an acquire memory ordering. This load will be referred to as `initial_invoke`.
    2.  If `initial_invoke` is NULL, the codeinst is not yet executable. `invoke` is NULL, `specsigflags` may be treated as 0b00, `specptr` may be treated as NULL.
    3.  Read the `specptr` field with at least an acquire memory ordering.
    4.  If `specptr` is NULL, then the `initial_invoke` pointer must not be relying on `specptr` to guarantee correct execution. Therefore, `invoke` is non-null, `specsigflags` may be treated as 0b00, `specptr` may be treated as NULL.
    5.  If `specptr` is non-null, then `initial_invoke` might not be the final `invoke` field that uses `specptr`. This can occur if `specptr` has been written, but `invoke` has not yet been written. Therefore, spin on the second bit of `specsigflags` until it is set to 1 with at least acquire memory ordering.
    6.  Re-read the `invoke` field with at least an acquire memory ordering. This load will be referred to as `final_invoke`.
    7.  Read the `specsigflags` field with any memory ordering.
    8.  `invoke` is `final_invoke`, `specsigflags` is the value read in step 7, `specptr` is the value read in step 3.
3.  When updating a `specptr` to a different but equivalent function pointer:


## [Execution](#Execution)
    1.  Perform a release store of the new function pointer to `specptr`. Races here must be benign, as the old function pointer is required to still be valid, and any new ones are also required to be valid as well. Once a pointer has been written to `specptr`, it must always be callable whether or not it is later overwritten.
Although these write, read, and update steps are complicated, they ensure that the JIT can update codeinsts without invalidating existing codeinsts, and that the JIT can update codeinsts without invalidating existing `invoke` pointers. This allows the JIT to potentially reoptimize functions at higher optimization levels in the future, and also will allow the JIT to support concurrent compilation of functions in the future.
-   [tdp](#citeref-tdp)This is not a totally-target independent pipeline, as transformations such as vectorization rely upon target information such as vector register width and cost modeling. Additionally, codegen itself makes a few target-dependent assumptions, and the optimization pipeline will take advantage of that knowledge.
------------------------------------------------------------------------


# Core.Builtins · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/builtins/


# [Core.Builtins](#lib-builtins)


## [Builtin Function APIs](#Builtin-Function-APIs)
The following Builtin function APIs are considered unstable, but provide the basic definitions for what defines the abilities and behaviors of a Julia program. They are typically accessed through a higher level generic API.
```julia
Core.memoryrefnew(::GenericMemory)
Core.memoryrefnew(::GenericMemoryRef, index::Int, [boundscheck::Bool])
```
Return a `GenericMemoryRef` for a `GenericMemory`. See [`memoryref`](../../base/arrays/#Core.memoryref).
This function requires Julia 1.11 or later.
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L25-L33)
```julia
Core..memoryrefoffset(::GenericMemoryRef)
```
Return the offset index that was used to construct the `MemoryRef`. See [`memoryref`](../../base/arrays/#Core.memoryref).
This function requires Julia 1.11 or later.
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L36-L43)
```julia
Core.memoryrefget(::GenericMemoryRef, ordering::Symbol, boundscheck::Bool)
```
Return the value stored at the `MemoryRef`, throwing a `BoundsError` if the `Memory` is empty. See `ref[]`. The memory ordering specified must be compatible with the `isatomic` parameter.
This function requires Julia 1.11 or later.
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L46-L54)
```julia
Core.memoryrefset!(::GenericMemoryRef, value, ordering::Symbol, boundscheck::Bool)
```


## [Builtin Function APIs](#Builtin-Function-APIs)
Store the value to the `MemoryRef`, throwing a `BoundsError` if the `Memory` is empty. See `ref[] = value`. The memory ordering specified must be compatible with the `isatomic` parameter.
This function requires Julia 1.11 or later.
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L57-L65)
```julia
Core.memoryref_isassigned(::GenericMemoryRef, ordering::Symbol, boundscheck::Bool)
```
Return whether there is a value stored at the `MemoryRef`, returning false if the `Memory` is empty. See [`isassigned(::Base.RefValue)`](../../base/c/#Base.isassigned-Tuple%7BBase.RefValue%7D), [`Core.memoryrefget`](#Core.memoryrefget). The memory ordering specified must be compatible with the `isatomic` parameter.
This function requires Julia 1.11 or later.
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L68-L77)
```julia
Core.memoryrefswap!(::GenericMemoryRef, value, ordering::Symbol, boundscheck::Bool)
```
Atomically perform the operations to simultaneously get and set a `MemoryRef` value.
This function requires Julia 1.11 or later.
See also [`swapproperty!`](../../base/base/#Base.swapproperty!) and [`Core.memoryrefset!`](#Core.memoryrefset!).
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L80-L89)
```julia
Core.memoryrefmodify!(::GenericMemoryRef, op, value, ordering::Symbol, boundscheck::Bool) -> Pair
```


## [Builtin Function APIs](#Builtin-Function-APIs)
Atomically perform the operations to get and set a `MemoryRef` value after applying the function `op`.
This function requires Julia 1.11 or later.
See also [`modifyproperty!`](../../base/base/#Base.modifyproperty!) and [`Core.memoryrefset!`](#Core.memoryrefset!).
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L92-L102)
```julia
Core.memoryrefreplace!(::GenericMemoryRef, expected, desired,
                       success_order::Symbol, fail_order::Symbol=success_order, boundscheck::Bool) -> (; old, success::Bool)
```
Atomically perform the operations to get and conditionally set a `MemoryRef` value.
This function requires Julia 1.11 or later.
See also [`replaceproperty!`](../../base/base/#Base.replaceproperty!) and [`Core.memoryrefset!`](#Core.memoryrefset!).
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L105-L115)
```julia
Core.memoryrefsetonce!(::GenericMemoryRef, value,
                       success_order::Symbol, fail_order::Symbol=success_order, boundscheck::Bool) -> success::Bool
```
Atomically perform the operations to set a `MemoryRef` to a given value, only if it was previously not set.
This function requires Julia 1.11 or later.
See also [`setpropertyonce!`](../../base/base/#Base.replaceproperty!) and [`Core.memoryrefset!`](#Core.memoryrefset!).
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L118-L129)


## [Builtin Function APIs](#Builtin-Function-APIs)
```julia
Core.Intrinsics.atomic_pointerref(pointer::Ptr{T}, order::Symbol) --> T
```
This function requires Julia 1.7 or later.
See [`unsafe_load`](../../base/c/#Base.unsafe_load).
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L132-L139)
```julia
Core.Intrinsics.atomic_pointerset(pointer::Ptr{T}, new::T, order::Symbol) --> pointer
```
This function requires Julia 1.7 or later.
See [`unsafe_store!`](../../base/c/#Base.unsafe_store!).
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L142-L149)
```julia
Core.Intrinsics.atomic_pointerswap(pointer::Ptr{T}, new::T, order::Symbol) --> old
```
This function requires Julia 1.7 or later.
See [`unsafe_swap!`](../../base/c/#Base.unsafe_swap!).
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L152-L159)
```julia
Core.Intrinsics.atomic_pointermodify(pointer::Ptr{T}, function::(old::T,arg::S)->T, arg::S, order::Symbol) --> old
```
This function requires Julia 1.7 or later.
See [`unsafe_modify!`](../../base/c/#Base.unsafe_modify!).
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L162-L169)
```julia
Core.Intrinsics.atomic_pointerreplace(pointer::Ptr{T}, expected::Any, new::T, success_order::Symbol, failure_order::Symbol) --> (old, cmp)
```
This function requires Julia 1.7 or later.


## [Builtin Function APIs](#Builtin-Function-APIs)
See [`unsafe_replace!`](../../base/c/#Base.unsafe_replace!).
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L172-L179)
```julia
Core.get_binding_type(module::Module, name::Symbol)
```
Retrieve the declared type of the binding `name` from the module `module`.
This function requires Julia 1.9 or later.
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/basedocs.jl#L2542-L2549)
```julia
Core.set_binding_type!(module::Module, name::Symbol, [type::Type])
```
Set the declared type of the binding `name` in the module `module` to `type`. Error if the binding already has a type that is not equivalent to `type`. If the `type` argument is absent, set the binding type to `Any` if unset, but do not error.
This function requires Julia 1.9 or later.
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/basedocs.jl#L2552-L2561)
```julia
Core.IntrinsicFunction <: Core.Builtin <: Function
```
The `Core.IntrinsicFunction` function define some basic primitives for what defines the abilities and behaviors of a Julia program
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L10-L15)
```julia
Core.Intrinsics
```
The `Core.Intrinsics` module holds the `Core.IntrinsicFunction` objects.
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L18-L22)


## [Builtin Function APIs](#Builtin-Function-APIs)
```julia
Core.IR
```
The `Core.IR` module exports the IR object model.
[source](https://github.com/JuliaLang/julia/blob/9615af0f269df4d371b8010e9507ed5bae86103b/base/docs/intrinsicsdocs.jl#L3-L7)
------------------------------------------------------------------------


# Fixing precompilation hangs due to open tasks or IO · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/precompile_hang/


# [Fixing precompilation hangs due to open tasks or IO](#Fixing-precompilation-hangs-due-to-open-tasks-or-IO)
On Julia 1.10 or higher, you might see the following message:
![Screenshot of precompilation hang](../img/precompilation_hang.png)
This may repeat. If it continues to repeat with no hints that it will resolve itself, you may have a "precompilation hang" that requires fixing. Even if it's transient, you might prefer to resolve it so that users will not be bothered by this warning. This page walks you through how to analyze and fix such issues.
If you follow the advice and hit `Ctrl-C`, you might see
```julia
^C Interrupted: Exiting precompilation...

  1 dependency had warnings during precompilation:
┌ Test1 [ac89d554-e2ba-40bc-bc5c-de68b658c982]
│  [pid 2745] waiting for IO to finish:
│   Handle type        uv_handle_t->data
│   timer              0x55580decd1e0->0x7f94c3a4c340
```
This message conveys two key pieces of information:
-   the hang is occurring during precompilation of `Test1`, a dependency of `Test2` (the package we were trying to load with `using Test2`)
-   during precompilation of `Test1`, Julia created a `Timer` object (use `?Timer` if you're unfamiliar with Timers) which is still open; until that closes, the process is hung
If this is enough of a hint for you to figure out how `timer = Timer(args...)` is being created, one good solution is to add `wait(timer)` if `timer` eventually finishes on its own, or `close(timer)` if you need to force-close it, before the final `end` of the module.


# [Fixing precompilation hangs due to open tasks or IO](#Fixing-precompilation-hangs-due-to-open-tasks-or-IO)
However, there are cases that may not be that straightforward. Usually the best option is to start by determining whether the hang is due to code in Test1 or whether it is due to one of Test1's dependencies:
-   Option 1: `Pkg.add("Aqua")` and use [`Aqua.test_persistent_tasks`](https://juliatesting.github.io/Aqua.jl/dev/#Aqua.test_persistent_tasks-Tuple%7BBase.PkgId%7D). This should help you identify which package is causing the problem, after which the instructions [below](#pchang_fix) should be followed. If needed, you can create a `PkgId` as `Base.PkgId(UUID("..."), "Test1")`, where `...` comes from the `uuid` entry in `Test1/Project.toml`.
-   Option 2: manually diagnose the source of the hang.
To manually diagnose:
1.  `Pkg.develop("Test1")`
2.  Comment out all the code `include`d or defined in `Test1`, *except* the `using/import` statements.
3.  Try `using Test2` (or even `using Test1` assuming that hangs too) again
Now we arrive at a fork in the road: either
-   the hang persists, indicating it is [due to one of your dependencies](#pchang_deps)
-   the hang disappears, indicating that it is [due to something in your code](#pchang_fix).


## [Diagnosing and fixing hangs due to a package dependency](#pchang_deps)
Use a binary search to identify the problematic dependency: start by commenting out half your dependencies, then when you isolate which half is responsible comment out half of that half, etc. (You don't have to remove them from the project, just comment out the `using`/`import` statements.)
Once you've identified a suspect (here we'll call it `ThePackageYouThinkIsCausingTheProblem`), first try precompiling that package. If it also hangs during precompilation, continue chasing the problem backwards.
However, most likely `ThePackageYouThinkIsCausingTheProblem` will precompile fine. This suggests it's in the function `ThePackageYouThinkIsCausingTheProblem.__init__`, which does not run during precompilation of `ThePackageYouThinkIsCausingTheProblem` but *does* in any package that loads `ThePackageYouThinkIsCausingTheProblem`. To test this theory, set up a minimal working example (MWE), something like
```julia
(@v1.10) pkg> generate MWE
  Generating  project MWE:
    MWE\Project.toml
    MWE\src\MWE.jl
```
where the source code of `MWE.jl` is
```julia
module MWE
using ThePackageYouThinkIsCausingTheProblem
end
```
and you've added `ThePackageYouThinkIsCausingTheProblem` to MWE's dependencies.


## [Diagnosing and fixing hangs due to a package dependency](#pchang_deps)
If that MWE reproduces the hang, you've found your culprit: `ThePackageYouThinkIsCausingTheProblem.__init__` must be creating the `Timer` object. If the timer object can be safely `close`d, that's a good option. Otherwise, the most common solution is to avoid creating the timer while *any* package is being precompiled: add
```julia
ccall(:jl_generating_output, Cint, ()) == 1 && return nothing
```
as the first line of `ThePackageYouThinkIsCausingTheProblem.__init__`, and it will avoid doing any initialization in any Julia process whose purpose is to precompile packages.


## [Fixing package code to avoid hangs](#pchang_fix)
Search your package for suggestive words (here like "Timer") and see if you can identify where the problem is being created. Note that a method *definition* like
```julia
maketimer() = Timer(timer -> println("hi"), 0; interval=1)
```
is not problematic in and of itself: it can cause this problem only if `maketimer` gets called while the module is being defined. This might be happening from a top-level statement such as
```julia
const GLOBAL_TIMER = maketimer()
```
or it might conceivably occur in a [precompile workload](https://github.com/JuliaLang/PrecompileTools.jl).
If you struggle to identify the causative lines, then consider doing a binary search: comment out sections of your package (or `include` lines to omit entire files) until you've reduced the problem in scope.
------------------------------------------------------------------------


# Reporting and analyzing crashes (segfaults) · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/backtraces/


# [Reporting and analyzing crashes (segfaults)](#Reporting-and-analyzing-crashes-(segfaults))
So you managed to break Julia. Congratulations! Collected here are some general procedures you can undergo for common symptoms encountered when something goes awry. Including the information from these debugging steps can greatly help the maintainers when tracking down a segfault or trying to figure out why your script is running slower than expected.
If you've been directed to this page, find the symptom that best matches what you're experiencing and follow the instructions to generate the debugging information requested. Table of symptoms:
-   [Segfaults during bootstrap (`sysimg.jl`)](#Segfaults-during-bootstrap-(sysimg.jl))
-   [Segfaults when running a script](#Segfaults-when-running-a-script)
-   [Errors during Julia startup](#Errors-during-Julia-startup)
-   [Other generic segfaults or unreachables reached](#Other-generic-segfaults-or-unreachables-reached)


## [Version/Environment info](#dev-version-info)
No matter the error, we will always need to know what version of Julia you are running. When Julia first starts up, a header is printed out with a version number and date. Please also include the output of `versioninfo()` (exported from the [`InteractiveUtils`](../../stdlib/InteractiveUtils/#InteractiveUtils.versioninfo) standard library) in any report you create:
```julia-repl
julia> using InteractiveUtils
julia> versioninfo()Julia Version 1.11.6
Commit 9615af0f269 (2025-07-09 12:58 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 128 × AMD EPYC 7502 32-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver2)
Threads: 1 default, 0 interactive, 1 GC (on 128 virtual cores)
Environment:
  JULIA_DEPOT_PATH = /cache/julia-buildkite-plugin/depots/018de947-b86e-4a96-a198-061223670bea
  JULIA_CPU_THREADS = 128
  JULIA_NUM_THREADS = 1
```


## [Segfaults during bootstrap (sysimg.jl)](#Segfaults-during-bootstrap-(sysimg.jl))
Segfaults toward the end of the `make` process of building Julia are a common symptom of something going wrong while Julia is preparsing the corpus of code in the `base/` folder. Many factors can contribute toward this process dying unexpectedly, however it is as often as not due to an error in the C-code portion of Julia, and as such must typically be debugged with a debug build inside of `gdb`. Explicitly:
Create a debug build of Julia:
```julia
$ cd <julia_root>
$ make debug
```
Note that this process will likely fail with the same error as a normal `make` incantation, however this will create a debug executable that will offer `gdb` the debugging symbols needed to get accurate backtraces. Next, manually run the bootstrap process inside of `gdb`:
```julia
$ cd base/
$ gdb -x ../contrib/debug_bootstrap.gdb
```
This will start `gdb`, attempt to run the bootstrap process using the debug build of Julia, and print out a backtrace if (when) it segfaults. You may need to hit `<enter>` a few times to get the full backtrace. Create a [gist](https://gist.github.com) with the backtrace, the [version info](#dev-version-info), and any other pertinent information you can think of and open a new [issue](https://github.com/JuliaLang/julia/issues?q=is%3Aopen) on Github with a link to the gist.


## [Segfaults when running a script](#Segfaults-when-running-a-script)
The procedure is very similar to [Segfaults during bootstrap (`sysimg.jl`)](#Segfaults-during-bootstrap-(sysimg.jl)). Create a debug build of Julia, and run your script inside of a debugged Julia process:
```julia
$ cd <julia_root>
$ make debug
$ gdb --args usr/bin/julia-debug <path_to_your_script>
```
Note that `gdb` will sit there, waiting for instructions. Type `r` to run the process, and `bt` to generate a backtrace once it segfaults:
```julia
(gdb) r
Starting program: /home/sabae/src/julia/usr/bin/julia-debug ./test.jl
...
(gdb) bt
```
Create a [gist](https://gist.github.com) with the backtrace, the [version info](#dev-version-info), and any other pertinent information you can think of and open a new [issue](https://github.com/JuliaLang/julia/issues?q=is%3Aopen) on Github with a link to the gist.


## [Errors during Julia startup](#Errors-during-Julia-startup)
Occasionally errors occur during Julia's startup process (especially when using binary distributions, as opposed to compiling from source) such as the following:
```julia
$ julia
exec: error -5
```
These errors typically indicate something is not getting loaded properly very early on in the bootup phase, and our best bet in determining what's going wrong is to use external tools to audit the disk activity of the `julia` process:
-   On Linux, use `strace`:
    ```julia
    $ strace julia
    ```
-   On OSX, use `dtruss`:
    ```julia
    $ dtruss -f julia
    ```
Create a [gist](https://gist.github.com) with the `strace`/ `dtruss` output, the [version info](#dev-version-info), and any other pertinent information and open a new [issue](https://github.com/JuliaLang/julia/issues?q=is%3Aopen) on Github with a link to the gist.


## [Other generic segfaults or unreachables reached](#Other-generic-segfaults-or-unreachables-reached)
As mentioned elsewhere, `julia` has good integration with `rr` for generating traces; this includes, on Linux, the ability to automatically run `julia` under `rr` and share the trace after a crash. This can be immensely helpful when debugging such crashes and is strongly encouraged when reporting crash issues to the JuliaLang/julia repo. To run `julia` under `rr` automatically, do:
```julia
julia --bug-report=rr
```
To generate the `rr` trace locally, but not share, you can do:
```julia
julia --bug-report=rr-local
```
Note that this is only works on Linux. The blog post on [Time Travelling Bug Reporting](https://julialang.org/blog/2020/05/rr/) has many more details.


## [Glossary](#Glossary)
A few terms have been used as shorthand in this guide:
-   `<julia_root>` refers to the root directory of the Julia source tree; e.g. it should contain folders such as `base`, `deps`, `src`, `test`, etc.....
------------------------------------------------------------------------


# gdb debugging tips · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/debuggingtips/


# [gdb debugging tips](#gdb-debugging-tips)


## [Displaying Julia variables](#Displaying-Julia-variables)
Within `gdb`, any `jl_value_t*` object `obj` can be displayed using
```julia
(gdb) call jl_(obj)
```
The object will be displayed in the `julia` session, not in the gdb session. This is a useful way to discover the types and values of objects being manipulated by Julia's C code.
Similarly, if you're debugging some of Julia's internals (e.g., `compiler.jl`), you can print `obj` using
```julia
ccall(:jl_, Cvoid, (Any,), obj)
```
This is a good way to circumvent problems that arise from the order in which julia's output streams are initialized.
Julia's flisp interpreter uses `value_t` objects; these can be displayed with `call fl_print(fl_ctx, ios_stdout, obj)`.


## [Useful Julia variables for Inspecting](#Useful-Julia-variables-for-Inspecting)
While the addresses of many variables, like singletons, can be useful to print for many failures, there are a number of additional variables (see `julia.h` for a complete list) that are even more useful.
-   (when in `jl_apply_generic`) `mfunc` and `jl_uncompress_ast(mfunc->def, mfunc->code)` :: for figuring out a bit about the call-stack
-   `jl_lineno` and `jl_filename` :: for figuring out what line in a test to go start debugging from (or figure out how far into a file has been parsed)
-   `$1` :: not really a variable, but still a useful shorthand for referring to the result of the last gdb command (such as `print`)
-   `jl_options` :: sometimes useful, since it lists all of the command line options that were successfully parsed
-   `jl_uv_stderr` :: because who doesn't like to be able to interact with stdio


## [Useful Julia functions for Inspecting those variables](#Useful-Julia-functions-for-Inspecting-those-variables)
-   `jl_print_task_backtraces(0)` :: Similar to gdb's `thread apply all bt` or lldb's `thread backtrace all`. Runs all threads while printing backtraces for all existing tasks.
-   `jl_gdblookup($pc)` :: For looking up the current function and line.
-   `jl_gdblookupinfo($pc)` :: For looking up the current method instance object.
-   `jl_gdbdumpcode(mi)` :: For dumping all of `code_typed/code_llvm/code_asm` when the REPL is not working right.
-   `jlbacktrace()` :: For dumping the current Julia backtrace stack to stderr. Only usable after `record_backtrace()` has been called.
-   `jl_dump_llvm_value(Value*)` :: For invoking `Value->dump()` in gdb, where it doesn't work natively. For example, `f->linfo->functionObject`, `f->linfo->specFunctionObject`, and `to_function(f->linfo)`.
-   `jl_dump_llvm_module(Module*)` :: For invoking `Module->dump()` in gdb, where it doesn't work natively.
-   `Type->dump()` :: only works in lldb. Note: add something like `;1` to prevent lldb from printing its prompt over the output
-   `jl_eval_string("expr")` :: for invoking side-effects to modify the current state or to lookup symbols
-   `jl_typeof(jl_value_t*)` :: for extracting the type tag of a Julia value (in gdb, call `macro define jl_typeof jl_typeof` first, or pick something short like `ty` for the first arg to define a shorthand)


## [Inserting breakpoints for inspection from gdb](#Inserting-breakpoints-for-inspection-from-gdb)
In your `gdb` session, set a breakpoint in `jl_breakpoint` like so:
```julia
(gdb) break jl_breakpoint
```
Then within your Julia code, insert a call to `jl_breakpoint` by adding
```julia
ccall(:jl_breakpoint, Cvoid, (Any,), obj)
```
where `obj` can be any variable or tuple you want to be accessible in the breakpoint.
It's particularly helpful to back up to the `jl_apply` frame, from which you can display the arguments to a function using, e.g.,
```julia
(gdb) call jl_(args[0])
```
Another useful frame is `to_function(jl_method_instance_t *li, bool cstyle)`. The `jl_method_instance_t*` argument is a struct with a reference to the final AST sent into the compiler. However, the AST at this point will usually be compressed; to view the AST, call `jl_uncompress_ast` and then pass the result to `jl_`:
```julia
#2  0x00007ffff7928bf7 in to_function (li=0x2812060, cstyle=false) at codegen.cpp:584
584          abort();
(gdb) p jl_(jl_uncompress_ast(li, li->ast))
```


## [Inserting breakpoints upon certain conditions](#Inserting-breakpoints-upon-certain-conditions)


### [Loading a particular file](#Loading-a-particular-file)
Let's say the file is `sysimg.jl`:
```julia
(gdb) break jl_load if strcmp(fname, "sysimg.jl")==0
```


### [Calling a particular method](#Calling-a-particular-method)
```julia
(gdb) break jl_apply_generic if strcmp((char*)(jl_symbol_name)(jl_gf_mtable(F)->name), "method_to_break")==0
```
Since this function is used for every call, you will make everything 1000x slower if you do this.


## [Dealing with signals](#Dealing-with-signals)
Julia requires a few signals to function properly. The profiler uses `SIGUSR2` for sampling and the garbage collector uses `SIGSEGV` for threads synchronization. If you are debugging some code that uses the profiler or multiple threads, you may want to let the debugger ignore these signals since they can be triggered very often during normal operations. The command to do this in GDB is (replace `SIGSEGV` with `SIGUSR2` or other signals you want to ignore):
```julia
(gdb) handle SIGSEGV noprint nostop pass
```
The corresponding LLDB command is (after the process is started):
```julia
(lldb) pro hand -p true -s false -n false SIGSEGV
```
If you are debugging a segfault with threaded code, you can set a breakpoint on `jl_critical_error` (`sigdie_handler` should also work on Linux and BSD) in order to only catch the actual segfault rather than the GC synchronization points.


## [Debugging during Julia's build process (bootstrap)](#Debugging-during-Julia's-build-process-(bootstrap))
Errors that occur during `make` need special handling. Julia is built in two stages, constructing `sys0` and `sys.ji`. To see what commands are running at the time of failure, use `make VERBOSE=1`.
At the time of this writing, you can debug build errors during the `sys0` phase from the `base` directory using:
```julia
julia/base$ gdb --args ../usr/bin/julia-debug -C native --build ../usr/lib/julia/sys0 sysimg.jl
```
You might need to delete all the files in `usr/lib/julia/` to get this to work.
You can debug the `sys.ji` phase using:
```julia
julia/base$ gdb --args ../usr/bin/julia-debug -C native --build ../usr/lib/julia/sys -J ../usr/lib/julia/sys0.ji sysimg.jl
```
By default, any errors will cause Julia to exit, even under gdb. To catch an error "in the act", set a breakpoint in `jl_error` (there are several other useful spots, for specific kinds of failures, including: `jl_too_few_args`, `jl_too_many_args`, and `jl_throw`).
Once an error is caught, a useful technique is to walk up the stack and examine the function by inspecting the related call to `jl_apply`. To take a real-world example:


## [Debugging during Julia's build process (bootstrap)](#Debugging-during-Julia's-build-process-(bootstrap))
```julia
Breakpoint 1, jl_throw (e=0x7ffdf42de400) at task.c:802
802 {
(gdb) p jl_(e)
ErrorException("auto_unbox: unable to determine argument type")
$2 = void
(gdb) bt 10
#0  jl_throw (e=0x7ffdf42de400) at task.c:802
#1  0x00007ffff65412fe in jl_error (str=0x7ffde56be000 <_j_str267> "auto_unbox:
   unable to determine argument type")
   at builtins.c:39
#2  0x00007ffde56bd01a in julia_convert_16886 ()
#3  0x00007ffff6541154 in jl_apply (f=0x7ffdf367f630, args=0x7fffffffc2b0, nargs=2) at julia.h:1281
...
```
The most recent `jl_apply` is at frame #3, so we can go back there and look at the AST for the function `julia_convert_16886`. This is the uniqued name for some method of `convert`. `f` in this frame is a `jl_function_t*`, so we can look at the type signature, if any, from the `specTypes` field:
```julia
(gdb) f 3
#3  0x00007ffff6541154 in jl_apply (f=0x7ffdf367f630, args=0x7fffffffc2b0, nargs=2) at julia.h:1281
1281            return f->fptr((jl_value_t*)f, args, nargs);
(gdb) p f->linfo->specTypes
$4 = (jl_tupletype_t *) 0x7ffdf39b1030
(gdb) p jl_( f->linfo->specTypes )
Tuple{Type{Float32}, Float64}           # <-- type signature for julia_convert_16886
```
Then, we can look at the AST for this function:


## [Debugging during Julia's build process (bootstrap)](#Debugging-during-Julia's-build-process-(bootstrap))
```julia
(gdb) p jl_( jl_uncompress_ast(f->linfo, f->linfo->ast) )
Expr(:lambda, Array{Any, 1}[:#s29, :x], Array{Any, 1}[Array{Any, 1}[], Array{Any, 1}[Array{Any, 1}[:#s29, :Any, 0], Array{Any, 1}[:x, :Any, 0]], Array{Any, 1}[], 0], Expr(:body,
Expr(:line, 90, :float.jl)::Any,
Expr(:return, Expr(:call, :box, :Float32, Expr(:call, :fptrunc, :Float32, :x)::Any)::Any)::Any)::Any)::Any
```
Finally, and perhaps most usefully, we can force the function to be recompiled in order to step through the codegen process. To do this, clear the cached `functionObject` from the `jl_lamdbda_info_t*`:
```julia
(gdb) p f->linfo->functionObject
$8 = (void *) 0x1289d070
(gdb) set f->linfo->functionObject = NULL
```
Then, set a breakpoint somewhere useful (e.g. `emit_function`, `emit_expr`, `emit_call`, etc.), and run codegen:
```julia
(gdb) p jl_compile(f)
... # your breakpoint here
```


## [Debugging precompilation errors](#Debugging-precompilation-errors)
Module precompilation spawns a separate Julia process to precompile each module. Setting a breakpoint or catching failures in a precompile worker requires attaching a debugger to the worker. The easiest approach is to set the debugger watch for new process launches matching a given name. For example:
```julia
(gdb) attach -w -n julia-debug
```
or:
```julia
(lldb) process attach -w -n julia-debug
```
Then run a script/command to start precompilation. As described earlier, use conditional breakpoints in the parent process to catch specific file-loading events and narrow the debugging window. (some operating systems may require alternative approaches, such as following each `fork` from the parent process)


## [Mozilla's Record and Replay Framework (rr)](#Mozilla's-Record-and-Replay-Framework-(rr))
Julia now works out of the box with [rr](https://rr-project.org/), the lightweight recording and deterministic debugging framework from Mozilla. This allows you to replay the trace of an execution deterministically. The replayed execution's address spaces, register contents, syscall data etc are exactly the same in every run.
A recent version of rr (3.1.0 or higher) is required.


### [Reproducing concurrency bugs with rr](#Reproducing-concurrency-bugs-with-rr)
rr simulates a single-threaded machine by default. In order to debug concurrent code you can use `rr record --chaos` which will cause rr to simulate between one to eight cores, chosen randomly. You might therefore want to set `JULIA_NUM_THREADS=8` and rerun your code under rr until you have caught your bug.
------------------------------------------------------------------------


# Using Valgrind with Julia · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/valgrind/


# [Using Valgrind with Julia](#Using-Valgrind-with-Julia)
[Valgrind](https://valgrind.org/) is a tool for memory debugging, memory leak detection, and profiling. This section describes things to keep in mind when using Valgrind to debug memory issues with Julia.


## [General considerations](#General-considerations)
By default, Valgrind assumes that there is no self modifying code in the programs it runs. This assumption works fine in most instances but fails miserably for a just-in-time compiler like `julia`. For this reason it is crucial to pass `--smc-check=all-non-file` to `valgrind`, else code may crash or behave unexpectedly (often in subtle ways).
In some cases, to better detect memory errors using Valgrind, it can help to compile `julia` with memory pools disabled. The compile-time flag `MEMDEBUG` disables memory pools in Julia, and `MEMDEBUG2` disables memory pools in FemtoLisp. To build `julia` with both flags, add the following line to `Make.user`:
```make
CFLAGS = -DMEMDEBUG -DMEMDEBUG2
```
Another thing to note: if your program uses multiple worker processes, it is likely that you want all such worker processes to run under Valgrind, not just the parent process. To do this, pass `--trace-children=yes` to `valgrind`.
Yet another thing to note: if using `valgrind` errors with `Unable to find compatible target in system image`, try rebuilding the sysimage with target `generic` or julia with `JULIA_CPU_TARGET=generic`.


## [Suppressions](#Suppressions)
Valgrind will typically display spurious warnings as it runs. To reduce the number of such warnings, it helps to provide a [suppressions file](https://valgrind.org/docs/manual/manual-core.html#manual-core.suppress) to Valgrind. A sample suppressions file is included in the Julia source distribution at `contrib/valgrind-julia.supp`.
The suppressions file can be used from the `julia/` source directory as follows:
```julia
$ valgrind --smc-check=all-non-file --suppressions=contrib/valgrind-julia.supp ./julia progname.jl
```
Any memory errors that are displayed should either be reported as bugs or contributed as additional suppressions. Note that some versions of Valgrind are [shipped with insufficient default suppressions](https://github.com/JuliaLang/julia/issues/8314#issuecomment-55766210), so that may be one thing to consider before submitting any bugs.


## [Running the Julia test suite under Valgrind](#Running-the-Julia-test-suite-under-Valgrind)
It is possible to run the entire Julia test suite under Valgrind, but it does take quite some time (typically several hours). To do so, run the following command from the `julia/test/` directory:
```julia
valgrind --smc-check=all-non-file --trace-children=yes --suppressions=$PWD/../contrib/valgrind-julia.supp ../julia runtests.jl all
```
If you would like to see a report of "definite" memory leaks, pass the flags `--leak-check=full --show-leak-kinds=definite` to `valgrind` as well.


## [Additional spurious warnings](#Additional-spurious-warnings)
This section covers Valgrind warnings that cannot be added to the suppressions file yet are nonetheless safe to ignore.


### [Unhandled rr system calls](#Unhandled-rr-system-calls)
Valgrind will emit a warning if it encounters any of the [system calls that are specific to rr](https://github.com/rr-debugger/rr/blob/master/src/preload/rrcalls.h), the [Record and Replay Framework](https://rr-project.org/). In particular, a warning about an unhandled `1008` syscall will be shown when julia tries to detect whether it is running under rr:
```julia
--xxxxxx-- WARNING: unhandled amd64-linux syscall: 1008
--xxxxxx-- You may be able to write your own handler.
--xxxxxx-- Read the file README_MISSING_SYSCALL_OR_IOCTL.
--xxxxxx-- Nevertheless we consider this a bug.  Please report
--xxxxxx-- it at http://valgrind.org/support/bug_reports.html.
```
This issue [has been reported](https://bugs.kde.org/show_bug.cgi?id=446401) to the Valgrind developers as they have requested.


## [Caveats](#Caveats)
Valgrind currently [does not support multiple rounding modes](https://bugs.kde.org/show_bug.cgi?id=136779), so code that adjusts the rounding mode will behave differently when run under Valgrind.
In general, if after setting `--smc-check=all-non-file` you find that your program behaves differently when run under Valgrind, it may help to pass `--tool=none` to `valgrind` as you investigate further. This will enable the minimal Valgrind machinery but will also run much faster than when the full memory checker is enabled.
------------------------------------------------------------------------


# External Profiler Support · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/external_profilers/


# [External Profiler Support](#External-Profiler-Support)
Julia provides explicit support for some external tracing profilers, enabling you to obtain a high-level overview of the runtime's execution behavior.
The currently supported profilers are:
-   [Tracy](https://github.com/wolfpld/tracy)
-   [Intel VTune (ITTAPI)](https://github.com/intel/ittapi)


### [Adding New Zones](#Adding-New-Zones)
To add new zones, use the `JL_TIMING` macro. You can find numerous examples throughout the codebase by searching for `JL_TIMING`. To add a new type of zone you add it to `JL_TIMING_OWNERS` (and possibly `JL_TIMING_EVENTS`).


### [Dynamically Enabling and Disabling Zones](#Dynamically-Enabling-and-Disabling-Zones)
The [`JULIA_TIMING_SUBSYSTEMS`](../../manual/environment-variables/#JULIA_TIMING_SUBSYSTEMS) environment variable allows you to enable or disable zones for a specific Julia run. For instance, setting the variable to `+GC,-INFERENCE` will enable the `GC` zones and disable the `INFERENCE` zones.


## [Tracy Profiler](#Tracy-Profiler)
[Tracy](https://github.com/wolfpld/tracy) is a flexible profiler that can be optionally integrated with Julia.
A typical Tracy session might look like this:
![Typical Tracy usage](../tracy.png)


### [Building Julia with Tracy](#Building-Julia-with-Tracy)
To enable Tracy integration, build Julia with the extra option `WITH_TRACY=1` in the `Make.user` file.


### [Installing the Tracy Profile Viewer](#Installing-the-Tracy-Profile-Viewer)
The easiest way to obtain the profile viewer is by adding the `TracyProfiler_jll` package and launching the profiler with:
```julia
run(TracyProfiler_jll.tracy())
```
On macOS, you may want to set the `TRACY_DPI_SCALE` environment variable to `1.0` if the UI elements in the profiler appear excessively large.
To run a "headless" instance that saves the trace to disk, use
```julia
run(`$(TracyProfiler_jll.capture()) -o mytracefile.tracy`)
```
instead.
For information on using the Tracy UI, refer to the Tracy manual.


### [Profiling Julia with Tracy](#Profiling-Julia-with-Tracy)
A typical workflow for profiling Julia with Tracy involves starting Julia using:
```julia
JULIA_WAIT_FOR_TRACY=1 ./julia -e '...'
```
The environment variable ensures that Julia waits until it has successfully connected to the Tracy profiler before continuing execution. Afterward, use the Tracy profiler UI, click `Connect`, and Julia execution should resume and profiling should start.


### [Profiling package precompilation with Tracy](#Profiling-package-precompilation-with-Tracy)
To profile a package precompilation process it is easiest to explicitly call into `Base.compilecache` with the package you want to precompile:
```julia
pkg = Base.identify_package("SparseArrays")
withenv("JULIA_WAIT_FOR_TRACY" => 1, "TRACY_PORT" => 9001) do
    Base.compilecache(pkg)
end
```
Here, we use a custom port for tracy which makes it easier to find the correct client in the Tracy UI to connect to.


### [Adding metadata to zones](#Adding-metadata-to-zones)
The various `jl_timing_show_*` and `jl_timing_printf` functions can be used to attach a string (or strings) to a zone. For example, the trace zone for inference shows the method instance that is being inferred.
The `TracyCZoneColor` function can be used to set the color of a certain zone. Search through the codebase to see how it is used.


### [Viewing Tracy files in your browser](#Viewing-Tracy-files-in-your-browser)
Visit https://topolarity.github.io/trace-viewer/ for an (experimental) web viewer for Tracy traces.
You can open a local `.tracy` file or provide a URL from the web (e.g. a file in a Github repo). If you load a trace file from the web, you can also share the page URL directly with others, enabling them to view the same trace.


### [Enabling stack trace samples](#Enabling-stack-trace-samples)
To enable call stack sampling in Tracy, build Julia with these options in your `Make.user` file:
```julia
WITH_TRACY := 1
WITH_TRACY_CALLSTACKS := 1
USE_BINARYBUILDER_LIBTRACYCLIENT := 0
```
You may also need to run `make -C deps clean-libtracyclient` to force a re-build of Tracy.
This feature has a significant impact on trace size and profiling overhead, so it is recommended to leave call stack sampling off when possible, especially if you intend to share your trace files online.
Note that the Julia JIT runtime does not yet have integration for Tracy's symbolification, so Julia functions will typically be unknown in these stack traces.


## [Intel VTune (ITTAPI) Profiler](#Intel-VTune-(ITTAPI)-Profiler)
*This section is yet to be written.*
------------------------------------------------------------------------


# Sanitizer support · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/sanitizers/


# [Sanitizer support](#Sanitizer-support)
[Sanitizers](https://github.com/google/sanitizers) can be used in custom Julia builds to make it easier to detect certain kinds of errors in Julia's internal C/C++ code.


## [Address Sanitizer: easy build](#Address-Sanitizer:-easy-build)
From a source-checkout of Julia, you should be able to build a version supporting address sanitization in Julia and LLVM as follows:
```sh
$ mkdir /tmp/julia
$ contrib/asan/build.sh /tmp/julia/
```
Here we've chosen `/tmp/julia` as a build directory, but you can choose whatever you wish. Once built, run the workload you wish to test with `/tmp/julia/julia`. Memory bugs will result in errors.
If you require customization or further detail, see the documentation below.


## [General considerations](#General-considerations)
Using Clang's sanitizers obviously requires you to use Clang (`USECLANG=1`), but there's another catch: most sanitizers require a run-time library, provided by the host compiler, while the instrumented code generated by Julia's JIT relies on functionality from that library. This implies that the LLVM version of your host compiler must match that of the LLVM library used within Julia.
An easy solution is to have a dedicated build folder for providing a matching toolchain, by building with `BUILD_LLVM_CLANG=1`. You can then refer to this toolchain from another build folder by specifying `USECLANG=1` while overriding the `CC` and `CXX` variables.
The sanitizers error out when they detect a shared library being opened using `RTLD_DEEPBIND` (ref: [google/sanitizers#611](https://github.com/google/sanitizers/issues/611)). Since [libblastrampoline](https://github.com/staticfloat/libblastrampoline) by default uses `RTLD_DEEPBIND`, we need to set the environment variable `LBT_USE_RTLD_DEEPBIND=0` when using a sanitizer.
To use one of of the sanitizers set `SANITIZE=1` and then the appropriate flag for the sanitizer you want to use.
On macOS, this might need some extra flags also to work. Altogether, it might look like this, plus one or more of the `SANITIZE_*` flags listed below:


## [General considerations](#General-considerations)
```julia
make -C deps USE_BINARYBUILDER_LLVM=0 LLVM_VER=svn stage-llvm

make -C src SANITIZE=1 USECLANG=1 \
    CC=~+/deps/scratch/llvm-svn/build_Release/bin/clang \
    CXX=~+/deps/scratch/llvm-svn/build_Release/bin/clang++ \
    CPPFLAGS="-isysroot $(xcode-select -p)/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk" \
    CXXFLAGS="-isystem $(xcode-select -p)/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1"
```
(or put these into your `Make.user`, so you don't need to remember them every time).


## [Address Sanitizer (ASAN)](#Address-Sanitizer-(ASAN))
For detecting or debugging memory bugs, you can use Clang's [address sanitizer (ASAN)](https://clang.llvm.org/docs/AddressSanitizer.html). By compiling with `SANITIZE_ADDRESS=1` you enable ASAN for the Julia compiler and its generated code. In addition, you can specify `LLVM_SANITIZE=1` to sanitize the LLVM library as well. Note that these options incur a high performance and memory cost. For example, using ASAN for Julia and LLVM makes `testall1` take 8-10 times as long while using 20 times as much memory (this can be reduced to respectively a factor of 3 and 4 by using the options described below).
By default, Julia sets the `allow_user_segv_handler=1` ASAN flag, which is required for signal delivery to work properly. You can define other options using the `ASAN_OPTIONS` environment flag, in which case you'll need to repeat the default option mentioned before. For example, memory usage can be reduced by specifying `fast_unwind_on_malloc=0` and `malloc_context_size=2`, at the cost of backtrace accuracy. For now, Julia also sets `detect_leaks=0`, but this should be removed in the future.


### [Example setup](#Example-setup)


#### [Step 1: Install toolchain](#Step-1:-Install-toolchain)
Checkout a Git worktree (or create out-of-tree build directory) at `$TOOLCHAIN_WORKTREE` and create a config file `$TOOLCHAIN_WORKTREE/Make.user` with
```julia
USE_BINARYBUILDER_LLVM=1
BUILD_LLVM_CLANG=1
```
Run:
```sh
cd $TOOLCHAIN_WORKTREE
make -C deps install-llvm install-clang install-llvm-tools
```
to install toolchain binaries in `$TOOLCHAIN_WORKTREE/usr/tools`


#### [Step 2: Build Julia with ASAN](#Step-2:-Build-Julia-with-ASAN)
Checkout a Git worktree (or create out-of-tree build directory) at `$BUILD_WORKTREE` and create a config file `$BUILD_WORKTREE/Make.user` with
```julia
TOOLCHAIN=$(TOOLCHAIN_WORKTREE)/usr/tools

# use our new toolchain
USECLANG=1
override CC=$(TOOLCHAIN)/clang
override CXX=$(TOOLCHAIN)/clang++
export ASAN_SYMBOLIZER_PATH=$(TOOLCHAIN)/llvm-symbolizer

USE_BINARYBUILDER_LLVM=1

override SANITIZE=1
override SANITIZE_ADDRESS=1

# make the GC use regular malloc/frees, which are hooked by ASAN
override WITH_GC_DEBUG_ENV=1

# default to a debug build for better line number reporting
override JULIA_BUILD_MODE=debug

# make ASAN consume less memory
export ASAN_OPTIONS=detect_leaks=0:fast_unwind_on_malloc=0:allow_user_segv_handler=1:malloc_context_size=2

JULIA_PRECOMPILE=1

# tell libblastrampoline to not use RTLD_DEEPBIND
export LBT_USE_RTLD_DEEPBIND=0
```
Run:
```sh
cd $BUILD_WORKTREE
make debug
```
to build `julia-debug` with ASAN.


## [Memory Sanitizer (MSAN)](#Memory-Sanitizer-(MSAN))
For detecting use of uninitialized memory, you can use Clang's [memory sanitizer (MSAN)](https://clang.llvm.org/docs/MemorySanitizer.html) by compiling with `SANITIZE_MEMORY=1`.


## [Thread Sanitizer (TSAN)](#Thread-Sanitizer-(TSAN))
For debugging data-races and other threading related issues you can use Clang's [thread sanitizer (TSAN)](https://clang.llvm.org/docs/ThreadSanitizer.html) by compiling with `SANITIZE_THREAD=1`.
------------------------------------------------------------------------


# Instrumenting Julia with DTrace, and bpftrace · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/probes/


# [Instrumenting Julia with DTrace, and bpftrace](#Instrumenting-Julia-with-DTrace,-and-bpftrace)
DTrace and bpftrace are tools that enable lightweight instrumentation of processes. You can turn the instrumentation on and off while the process is running, and with instrumentation off the overhead is minimal.
Support for probes was added in Julia 1.8
This documentation has been written from a Linux perspective, most of this should hold on Mac OS/Darwin and FreeBSD.


## [Enabling support](#Enabling-support)
On Linux install the `systemtap` package that has a version of `dtrace` and create a `Make.user` file containing
```julia
WITH_DTRACE=1
```
to enable USDT probes.


### [Verifying](#Verifying)
```julia
> readelf -n usr/lib/libjulia-internal.so.1

Displaying notes found in: .note.gnu.build-id
  Owner                Data size  Description
  GNU                  0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: 57161002f35548772a87418d2385c284ceb3ead8

Displaying notes found in: .note.stapsdt
  Owner                Data size  Description
  stapsdt              0x00000029 NT_STAPSDT (SystemTap probe descriptors)
    Provider: julia
    Name: gc__begin
    Location: 0x000000000013213e, Base: 0x00000000002bb4da, Semaphore: 0x0000000000346cac
    Arguments:
  stapsdt              0x00000032 NT_STAPSDT (SystemTap probe descriptors)
    Provider: julia
    Name: gc__stop_the_world
    Location: 0x0000000000132144, Base: 0x00000000002bb4da, Semaphore: 0x0000000000346cae
    Arguments:
  stapsdt              0x00000027 NT_STAPSDT (SystemTap probe descriptors)
    Provider: julia
    Name: gc__end
    Location: 0x000000000013214a, Base: 0x00000000002bb4da, Semaphore: 0x0000000000346cb0
    Arguments:
  stapsdt              0x0000002d NT_STAPSDT (SystemTap probe descriptors)
    Provider: julia
    Name: gc__finalizer
    Location: 0x0000000000132150, Base: 0x00000000002bb4da, Semaphore: 0x0000000000346cb2
    Arguments:
```


## [Adding probes in libjulia](#Adding-probes-in-libjulia)
Probes are declared in dtraces format in the file `src/uprobes.d`. The generated header file is included in `src/julia_internal.h` and if you add probes you should provide a noop implementation there.
The header will contain a semaphore `*_ENABLED` and the actual call to the probe. If the probe arguments are expensive to compute you should first check if the probe is enabled and then compute the arguments and call the probe.
```c
  if (JL_PROBE_{PROBE}_ENABLED())
    auto expensive_arg = ...;
    JL_PROBE_{PROBE}(expensive_arg);
```
If your probe has no arguments it is preferred to not include the semaphore check. With USDT probes enabled the cost of a semaphore is a memory load, irrespective of the fact that the probe is enabled or not.
```c
#define JL_PROBE_GC_BEGIN_ENABLED() __builtin_expect (julia_gc__begin_semaphore, 0)
__extension__ extern unsigned short julia_gc__begin_semaphore __attribute__ ((unused)) __attribute__ ((section (".probes")));
```
Whereas the probe itself is a noop sled that will be patched to a trampoline to the probe handler.


## [Available probes](#Available-probes)


### [GC probes](#GC-probes)
1.  `julia:gc__begin`: GC begins running on one thread and triggers stop-the-world.
2.  `julia:gc__stop_the_world`: All threads have reached a safepoint and GC runs.
3.  `julia:gc__mark__begin`: Beginning the mark phase
4.  `julia:gc__mark_end(scanned_bytes, perm_scanned)`: Mark phase ended
5.  `julia:gc__sweep_begin(full)`: Starting sweep
6.  `julia:gc__sweep_end`: Sweep phase finished
7.  `julia:gc__end`: GC is finished, other threads continue work
8.  `julia:gc__finalizer`: Initial GC thread has finished running finalizers


### [Task runtime probes](#Task-runtime-probes)
1.  `julia:rt__run__task(task)`: Switching to task `task` on current thread.
2.  `julia:rt__pause__task(task)`: Switching from task `task` on current thread.
3.  `julia:rt__new__task(parent, child)`: Task `parent` created task `child` on current thread.
4.  `julia:rt__start__task(task)`: Task `task` started for the first time with a new stack.
5.  `julia:rt__finish__task(task)`: Task `task` finished and will no longer execute.
6.  `julia:rt__start__process__events(task)`: Task `task` started processing libuv events.
7.  `julia:rt__finish__process__events(task)`: Task `task` finished processing libuv events.


### [Task queue probes](#Task-queue-probes)
1.  `julia:rt__taskq__insert(ptls, task)`: Thread `ptls` attempted to insert `task` into a PARTR multiq.
2.  `julia:rt__taskq__get(ptls, task)`: Thread `ptls` popped `task` from a PARTR multiq.


### [Thread sleep/wake probes](#Thread-sleep/wake-probes)
1.  `julia:rt__sleep__check__wake(ptls, old_state)`: Thread (PTLS `ptls`) waking up, previously in state `old_state`.
2.  `julia:rt__sleep__check__wakeup(ptls)`: Thread (PTLS `ptls`) woke itself up.
3.  `julia:rt__sleep__check__sleep(ptls)`: Thread (PTLS `ptls`) is attempting to sleep.
4.  `julia:rt__sleep__check__taskq__wake(ptls)`: Thread (PTLS `ptls`) fails to sleep due to tasks in PARTR multiq.
5.  `julia:rt__sleep__check__task__wake(ptls)`: Thread (PTLS `ptls`) fails to sleep due to tasks in Base workqueue.
6.  `julia:rt__sleep__check__uv__wake(ptls)`: Thread (PTLS `ptls`) fails to sleep due to libuv wakeup.


## [Probe usage examples](#Probe-usage-examples)


### [GC stop-the-world latency](#GC-stop-the-world-latency)
An example `bpftrace` script is given in `contrib/gc_stop_the_world_latency.bt` and it creates a histogram of the latency for all threads to reach a safepoint.
Running this Julia code, with `julia -t 2`
```julia
using Base.Threads

fib(x) = x <= 1 ? 1 : fib(x-1) + fib(x-2)

beaver = @spawn begin
    while true
        fib(30)
        # A manual safepoint is necessary since otherwise this loop
        # may never yield to GC.
        GC.safepoint()
    end
end

allocator = @spawn begin
    while true
        zeros(1024)
    end
end

wait(allocator)
```
and in a second terminal
```julia
> sudo contrib/bpftrace/gc_stop_the_world_latency.bt
Attaching 4 probes...
Tracing Julia GC Stop-The-World Latency... Hit Ctrl-C to end.
^C

@usecs[1743412]:
[4, 8)               971 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8, 16)              837 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@        |
[16, 32)             129 |@@@@@@                                              |
[32, 64)              10 |                                                    |
[64, 128)              1 |                                                    |
```
We can see the latency distribution of the stop-the-world phase in the executed Julia process.


### [Task spawn monitor](#Task-spawn-monitor)
It's sometimes useful to know when a task is spawning other tasks. This is very easy to see with `rt__new__task`. The first argument to the probe, `parent`, is the existing task which is creating a new task. This means that if you know the address of the task you want to monitor, you can easily just look at the tasks that that specific task spawned. Let's see how to do this; first let's start a Julia session and get the PID and REPL's task address:
```julia
> julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.2 (2021-07-14)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

1> getpid()
997825

2> current_task()
Task (runnable) @0x00007f524d088010
```
Now we can start `bpftrace` and have it monitor `rt__new__task` for *only* this parent:
`sudo bpftrace -p 997825 -e 'usdt:usr/lib/libjulia-internal.so:julia:rt__new__task /arg0==0x00007f524d088010/{ printf("Task: %x\n", arg0); }'`
(Note that in the above, `arg0` is the first argument, `parent`).
And if we spawn a single task:
`@async 1+1`
we see this task being created:
`Task: 4d088010`
However, if we spawn a bunch of tasks from that newly-spawned task:
```julia
@async for i in 1:10
   @async 1+1
end
```
we still only see one task from `bpftrace`:
`Task: 4d088010`


### [Task spawn monitor](#Task-spawn-monitor)
and it's still the same task we were monitoring! Of course, we can remove this filter to see *all* newly-created tasks just as easily:
`sudo bpftrace -p 997825 -e 'usdt:usr/lib/libjulia-internal.so:julia:rt__new__task { printf("Task: %x\n", arg0); }'`
```julia
Task: 4d088010
Task: 4dc4e290
Task: 4dc4e290
Task: 4dc4e290
Task: 4dc4e290
Task: 4dc4e290
Task: 4dc4e290
Task: 4dc4e290
Task: 4dc4e290
Task: 4dc4e290
Task: 4dc4e290
```
We can see our root task, and the newly-spawned task as the parent of the ten even newer tasks.


### [Thundering herd detection](#Thundering-herd-detection)
Task runtimes can often suffer from the "thundering herd" problem: when some work is added to a quiet task runtime, all threads may be woken up from their slumber, even if there isn't enough work for each thread to process. This can cause extra latency and CPU cycles while all threads awaken (and simultaneously go back to sleep, not finding any work to execute).
We can see this problem illustrated with `bpftrace` quite easily. First, in one terminal we start Julia with multiple threads (6 in this example), and get the PID of that process:
```julia
> julia -t 6
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.2 (2021-07-14)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

1> getpid()
997825
```
And in another terminal we start `bpftrace` monitoring our process, specifically probing the `rt__sleep__check__wake` hook:
`sudo bpftrace -p 997825 -e 'usdt:usr/lib/libjulia-internal.so:julia:rt__sleep__check__wake { printf("Thread wake up! %x\n", arg0); }'`
Now, we create and execute a single task in Julia:
`Threads.@spawn 1+1`
And in `bpftrace` we see printed out something like:
```julia
Thread wake up! 3f926100
Thread wake up! 3ebd5140
Thread wake up! 3f876130
Thread wake up! 3e2711a0
Thread wake up! 3e312190
```


### [Thundering herd detection](#Thundering-herd-detection)
Even though we only spawned a single task (which only one thread could process at a time), we woke up all of our other threads! In the future, a smarter task runtime might only wake up a single thread (or none at all; the spawning thread could execute this task!), and we should see this behavior go away.


### [Task Monitor with BPFnative.jl](#Task-Monitor-with-BPFnative.jl)
BPFnative.jl is able to attach to USDT probe points just like `bpftrace`. There is a demo available for monitoring the task runtime, GC, and thread sleep/wake transitions [here](https://github.com/jpsamaroo/BPFnative.jl/blob/master/examples/task-runtime.jl).


## [Notes on using bpftrace](#Notes-on-using-bpftrace)
An example probe in the bpftrace format looks like:
```julia
usdt:usr/lib/libjulia-internal.so:julia:gc__begin
{
  @start[pid] = nsecs;
}
```
The probe declaration takes the kind `usdt`, then either the path to the library or the PID, the provider name `julia` and the probe name `gc__begin`. Note that I am using a relative path to the `libjulia-internal.so`, but this might need to be an absolute path on a production system.


## [Useful references:](#Useful-references:)
-   [Julia Evans blog on Linux tracing systems](https://jvns.ca/blog/2017/07/05/linux-tracing-systems)
-   [LWN article on USDT and BPF](https://lwn.net/Articles/753601/)
-   [GDB support for probes](https://sourceware.org/gdb/onlinedocs/gdb/Static-Probe-Points.html)
-   [Brendan Gregg – Linux Performance](https://www.brendangregg.com/linuxperf.html)
------------------------------------------------------------------------


# Building Julia (Detailed) · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/build/build/


# [Building Julia (Detailed)](#Building-Julia-(Detailed))


## [Downloading the Julia source code](#Downloading-the-Julia-source-code)
If you are behind a firewall, you may need to use the `https` protocol instead of the `git` protocol:
```sh
git config --global url."https://".insteadOf git://
```
Be sure to also configure your system to use the appropriate proxy settings, e.g. by setting the `https_proxy` and `http_proxy` variables.


## [Building Julia](#Building-Julia)
When compiled the first time, the build will automatically download pre-built [external dependencies](#Required-Build-Tools-and-External-Libraries). If you prefer to build all the dependencies on your own, or are building on a system that cannot access the network during the build process, add the following in `Make.user`:
```julia
USE_BINARYBUILDER=0
```
Building Julia requires 5GiB if building all dependencies and approximately 4GiB of virtual memory.
To perform a parallel build, use `make -j N` and supply the maximum number of concurrent processes. If the defaults in the build do not work for you, and you need to set specific make parameters, you can save them in `Make.user`, and place the file in the root of your Julia source. The build will automatically check for the existence of `Make.user` and use it if it exists.
You can create out-of-tree builds of Julia by specifying `make O=<build-directory> configure` on the command line. This will create a directory mirror, with all of the necessary Makefiles to build Julia, in the specified directory. These builds will share the source files in Julia and `deps/srccache`. Each out-of-tree build directory can have its own `Make.user` file to override the global `Make.user` file in the top-level folder.


## [Building Julia](#Building-Julia)
If everything works correctly, you will see a Julia banner and an interactive prompt into which you can enter expressions for evaluation. (Errors related to libraries might be caused by old, incompatible libraries sitting around in your PATH. In this case, try moving the `julia` directory earlier in the PATH). Note that most of the instructions above apply to unix systems.
To run julia from anywhere you can:
-   add an alias (in `bash`: `echo "alias julia='/path/to/install/folder/bin/julia'" >> ~/.bashrc && source ~/.bashrc`), or
-   add a soft link to the `julia` executable in the `julia` directory to `/usr/local/bin` (or any suitable directory already in your path), or
-   add the `julia` directory to your executable path for this shell session (in `bash`: `export PATH="$(pwd):$PATH"` ; in `csh` or `tcsh`:
`set path= ( $path $cwd )` ), or
-   add the `julia` directory to your executable path permanently (e.g. in `.bash_profile`), or
-   write `prefix=/path/to/install/folder` into `Make.user` and then run `make install`. If there is a version of Julia already installed in this folder, you should delete it before running `make install`.
Some of the options you can set to control the build of Julia are listed and documented at the beginning of the file `Make.inc`, but you should never edit it for this purpose, use `Make.user` instead.
Julia's Makefiles define convenient automatic rules called `print-<VARNAME>` for printing the value of variables, replacing `<VARNAME>` with the name of the variable to print the value of. For example


## [Building Julia](#Building-Julia)
```console
$ make print-JULIA_PRECOMPILE
JULIA_PRECOMPILE=1
```
These rules are useful for debugging purposes.
Now you should be able to run Julia like this:
```julia
julia
```
If you are building a Julia package for distribution on Linux, macOS, or Windows, take a look at the detailed notes in [distributing.md](https://github.com/JuliaLang/julia/blob/master/doc/src/devdocs/build/distributing.md).


## [Updating an existing source tree](#Updating-an-existing-source-tree)
If you have previously downloaded `julia` using `git clone`, you can update the existing source tree using `git pull` rather than starting anew:
```sh
cd julia
git pull && make
```
Assuming that you had made no changes to the source tree that will conflict with upstream updates, these commands will trigger a build to update to the latest version.


## [General troubleshooting](#General-troubleshooting)
1.  Over time, the base library may accumulate enough changes such that the bootstrapping process in building the system image will fail. If this happens, the build may fail with an error like
    ```sh
     *** This error is usually fixed by running 'make clean'. If the error persists, try 'make cleanall' ***
    ```
    As described, running `make clean && make` is usually sufficient. Occasionally, the stronger cleanup done by `make cleanall` is needed.
2.  New versions of external dependencies may be introduced which may occasionally cause conflicts with existing builds of older versions.
    a\. Special `make` targets exist to help wipe the existing build of a dependency. For example, `make -C deps clean-llvm` will clean out the existing build of `llvm` so that `llvm` will be rebuilt from the downloaded source distribution the next time `make` is called. `make -C deps distclean-llvm` is a stronger wipe which will also delete the downloaded source distribution, ensuring that a fresh copy of the source distribution will be downloaded and that any new patches will be applied the next time `make` is called.
    b\. To delete existing binaries of `julia` and all its dependencies, delete the `./usr` directory *in the source tree*.
3.  If you've updated macOS recently, be sure to run `xcode-select --install` to update the command line tools. Otherwise, you could run into errors for missing headers and libraries, such as `ld: library not found for -lcrt1.10.6.o`.


## [General troubleshooting](#General-troubleshooting)
4.  If you've moved the source directory, you might get errors such as `CMake Error: The current CMakeCache.txt directory ... is different than the directory ... where     CMakeCache.txt was created.`, in which case you may delete the offending dependency under `deps`
5.  In extreme cases, you may wish to reset the source tree to a pristine state. The following git commands may be helpful:
    ```sh
     git reset --hard #Forcibly remove any changes to any files under version control
     git clean -x -f -d #Forcibly remove any file or directory not under version control
    ```
    *To avoid losing work, make sure you know what these commands do before you run them. `git` will not be able to undo these changes!*


## [Platform-Specific Notes](#Platform-Specific-Notes)
Notes for various operating systems:
-   [Linux](https://github.com/JuliaLang/julia/blob/master/doc/src/devdocs/build/linux.md)
-   [macOS](https://github.com/JuliaLang/julia/blob/master/doc/src/devdocs/build/macos.md)
-   [Windows](https://github.com/JuliaLang/julia/blob/master/doc/src/devdocs/build/windows.md)
-   [FreeBSD](https://github.com/JuliaLang/julia/blob/master/doc/src/devdocs/build/freebsd.md)
Notes for various architectures:
-   [ARM](https://github.com/JuliaLang/julia/blob/master/doc/src/devdocs/build/arm.md)


## [Required Build Tools and External Libraries](#Required-Build-Tools-and-External-Libraries)
Building Julia requires that the following software be installed:
-   **\[GNU make\]** — building dependencies.
-   **\[gcc & g++\]\[gcc\]** (>= 7.1) or **\[Clang\]\[clang\]** (>= 5.0, \>= 9.3 for Apple Clang) — compiling and linking C, C++.
-   **\[libatomic\]\[gcc\]** — provided by **\[gcc\]** and needed to support atomic operations.
-   **\[python\]** (>=2.7) — needed to build LLVM.
-   **\[gfortran\]** — compiling and linking Fortran libraries.
-   **\[perl\]** — preprocessing of header files of libraries.
-   **\[wget\]**, **\[curl\]**, or **\[fetch\]** (FreeBSD) — to automatically download external libraries.
-   **\[m4\]** — needed to build GMP.
-   **\[awk\]** — helper tool for Makefiles.
-   **\[patch\]** — for modifying source code.
-   **\[cmake\]** (>= 3.4.3) — needed to build `libgit2`.
-   **\[pkg-config\]** — needed to build `libgit2` correctly, especially for proxy support.
-   **\[powershell\]** (>= 3.0) — necessary only on Windows.
-   **\[which\]** — needed for checking build dependencies.
On Debian-based distributions (e.g. Ubuntu), you can easily install them with `apt-get`:
```julia
sudo apt-get install build-essential libatomic1 python gfortran perl wget m4 cmake pkg-config curl
```


## [Required Build Tools and External Libraries](#Required-Build-Tools-and-External-Libraries)
Julia uses the following external libraries, which are automatically downloaded (or in a few cases, included in the Julia source repository) and then compiled from source the first time you run `make`. The specific version numbers of these libraries that Julia uses are listed in [`deps/$(libname).version`](https://github.com/JuliaLang/julia/blob/master/deps/):
-   **\[LLVM\]** (15.0 + [patches](https://github.com/JuliaLang/llvm-project/tree/julia-release/15.x)) — compiler infrastructure (see [note below](#llvm)).
-   **\[FemtoLisp\]** — packaged with Julia source, and used to implement the compiler front-end.
-   **\[libuv\]** (custom fork) — portable, high-performance event-based I/O library.
-   **\[OpenLibm\]** — portable libm library containing elementary math functions.
-   **\[DSFMT\]** — fast Mersenne Twister pseudorandom number generator library.
-   **\[OpenBLAS\]** — fast, open, and maintained \[basic linear algebra subprograms (BLAS)\]
-   **\[LAPACK\]** — library of linear algebra routines for solving systems of simultaneous linear equations, least-squares solutions of linear systems of equations, eigenvalue problems, and singular value problems.
-   **\[MKL\]** (optional) – OpenBLAS and LAPACK may be replaced by Intel's MKL library.
-   **\[SuiteSparse\]** — library of linear algebra routines for sparse matrices.
-   **\[PCRE\]** — Perl-compatible regular expressions library.
-   **\[GMP\]** — GNU multiple precision arithmetic library, needed for `BigInt` support.


## [Required Build Tools and External Libraries](#Required-Build-Tools-and-External-Libraries)
-   **\[MPFR\]** — GNU multiple precision floating point library, needed for arbitrary precision floating point (`BigFloat`) support.
-   **\[libgit2\]** — Git linkable library, used by Julia's package manager.
-   **\[curl\]** — libcurl provides download and proxy support.
-   **\[libssh2\]** — library for SSH transport, used by libgit2 for packages with SSH remotes.
-   **\[mbedtls\]** — library used for cryptography and transport layer security, used by libssh2
-   **\[utf8proc\]** — a library for processing UTF-8 encoded Unicode strings.
-   **\[LLVM libunwind\]** — LLVM's fork of \[libunwind\], a library that determines the call-chain of a program.
-   **\[ITTAPI\]** — Intel's Instrumentation and Tracing Technology and Just-In-Time API.


## [Required Build Tools and External Libraries](#Required-Build-Tools-and-External-Libraries)
\[GNU make\]: https://www.gnu.org/software/make \[patch\]: https://www.gnu.org/software/patch \[wget\]: https://www.gnu.org/software/wget \[m4\]: https://www.gnu.org/software/m4 \[awk\]: https://www.gnu.org/software/gawk \[gcc\]: https://gcc.gnu.org \[clang\]: https://clang.llvm.org \[python\]: https://www.python.org/ \[gfortran\]: https://gcc.gnu.org/fortran/ \[curl\]: https://curl.haxx.se \[fetch\]: https://www.freebsd.org/cgi/man.cgi?fetch(1) \[perl\]: https://www.perl.org \[cmake\]: https://www.cmake.org \[OpenLibm\]: https://github.com/JuliaLang/openlibm \[DSFMT\]: https://github.com/MersenneTwister-Lab/dSFMT \[OpenBLAS\]: https://github.com/xianyi/OpenBLAS \[LAPACK\]: https://www.netlib.org/lapack \[MKL\]: https://software.intel.com/en-us/articles/intel-mkl \[SuiteSparse\]: https://people.engr.tamu.edu/davis/suitesparse.html \[PCRE\]: https://www.pcre.org \[LLVM\]: https://www.llvm.org \[LLVM libunwind\]: https://github.com/llvm/llvm-project/tree/main/libunwind \[FemtoLisp\]: https://github.com/JeffBezanson/femtolisp \[GMP\]: https://gmplib.org \[MPFR\]: https://www.mpfr.org \[libuv\]: https://github.com/JuliaLang/libuv \[libgit2\]: https://libgit2.org/ \[utf8proc\]: https://julialang.org/utf8proc/ \[libunwind\]: https://www.nongnu.org/libunwind \[libssh2\]: https://www.libssh2.org \[mbedtls\]: https://tls.mbed.org/ \[pkg-config\]: https://www.freedesktop.org/wiki/Software/pkg-config/ \[powershell\]: https://docs.microsoft.com/en-us/powershell/scripting/wmf/overview \[which\]: https://carlowood.github.io/which/ \[ITTAPI\]: https://github.com/intel/ittapi


## [Build dependencies](#Build-dependencies)
If you already have one or more of these packages installed on your system, you can prevent Julia from compiling duplicates of these libraries by passing `USE_SYSTEM_...=1` to `make` or adding the line to `Make.user`. The complete list of possible flags can be found in `Make.inc`.
Please be aware that this procedure is not officially supported, as it introduces additional variability into the installation and versioning of the dependencies, and is recommended only for system package maintainers. Unexpected compile errors may result, as the build system will do no further checking to ensure the proper packages are installed.


### [LLVM](#LLVM)
The most complicated dependency is LLVM, for which we require additional patches from upstream (LLVM is not backward compatible).
For packaging Julia with LLVM, we recommend either:
-   bundling a Julia-only LLVM library inside the Julia package, or
-   adding the patches to the LLVM package of the distribution.
    -   A complete list of patches is available in on [Github](https://github.com/JuliaLang/llvm-project) see the `julia-release/15.x` branch.
    -   The only Julia-specific patch is the lib renaming (`llvm7-symver-jlprefix.patch`), which should *not* be applied to a system LLVM.
    -   The remaining patches are all upstream bug fixes, and have been contributed into upstream LLVM.
Using an unpatched or different version of LLVM will result in errors and/or poor performance. You can build a different version of LLVM from a remote Git repository with the following options in the `Make.user` file:


### [LLVM](#LLVM)
```make
# Force source build of LLVM
USE_BINARYBUILDER_LLVM = 0
# Use Git for fetching LLVM source code
# this is either `1` to get all of them
DEPS_GIT = 1
# or a space-separated list of specific dependencies to download with git
DEPS_GIT = llvm

# Other useful options:
#URL of the Git repository you want to obtain LLVM from:
#  LLVM_GIT_URL = ...
#Name of the alternate branch to clone from git
#  LLVM_BRANCH = julia-16.0.6-0
#SHA hash of the alterate commit to check out automatically
#  LLVM_SHA1 = $(LLVM_BRANCH)
#List of LLVM targets to build.  It is strongly recommended to keep at least all the
#default targets listed in `deps/llvm.mk`, even if you don't necessarily need all of them.
#  LLVM_TARGETS = ...
#Use ccache for faster recompilation in case you need to restart a build.
#  USECCACHE = 1
#  CMAKE_GENERATOR=Ninja
#  LLVM_ASSERTIONS=1
#  LLVM_DEBUG=Symbols
```
The various build phases are controlled by specific files:
-   `deps/llvm.version` : touch or change to checkout a new version, `make get-llvm check-llvm`
-   `deps/srccache/llvm/source-extracted` : result of `make extract-llvm`
-   `deps/llvm/build_Release*/build-configured` : result of `make configure-llvm`
-   `deps/llvm/build_Release*/build-configured` : result of `make compile-llvm`
-   `usr-staging/llvm/build_Release*.tgz` : result of `make stage-llvm` (regenerate with `make reinstall-llvm`)
-   `usr/manifest/llvm` : result of `make install-llvm` (regenerate with `make uninstall-llvm`)
-   `make version-check-llvm` : runs every time to warn the user if there are local modifications


### [LLVM](#LLVM)
Though Julia can be built with newer LLVM versions, support for this should be regarded as experimental and not suitable for packaging.


### [libuv](#libuv)
Julia uses a custom fork of libuv. It is a small dependency, and can be safely bundled in the same package as Julia, and will not conflict with the system library. Julia builds should *not* try to use the system libuv.


### [BLAS and LAPACK](#BLAS-and-LAPACK)
As a high-performance numerical language, Julia should be linked to a multi-threaded BLAS and LAPACK, such as OpenBLAS or ATLAS, which will provide much better performance than the reference `libblas` implementations which may be default on some systems.


## [Source distributions of releases](#Source-distributions-of-releases)
Each pre-release and release of Julia has a "full" source distribution and a "light" source distribution.
The full source distribution contains the source code for Julia and all dependencies so that it can be built from source without an internet connection. The light source distribution does not include the source code of dependencies.
For example, `julia-1.0.0.tar.gz` is the light source distribution for the `v1.0.0` release of Julia, while `julia-1.0.0-full.tar.gz` is the full source distribution.


## [Building Julia from source with a Git checkout of a stdlib](#Building-Julia-from-source-with-a-Git-checkout-of-a-stdlib)
If you need to build Julia from source with a Git checkout of a stdlib, then use `make DEPS_GIT=NAME_OF_STDLIB` when building Julia.
For example, if you need to build Julia from source with a Git checkout of Pkg, then use `make DEPS_GIT=Pkg` when building Julia. The `Pkg` repo is in `stdlib/Pkg`, and created initially with a detached `HEAD`. If you're doing this from a pre-existing Julia repository, you may need to `make clean` beforehand.
If you need to build Julia from source with Git checkouts of more than one stdlib, then `DEPS_GIT` should be a space-separated list of the stdlib names. For example, if you need to build Julia from source with a Git checkout of Pkg, Tar, and Downloads, then use `make DEPS_GIT='Pkg Tar Downloads'` when building Julia.


## [Building an "assert build" of Julia](#Building-an-%22assert-build%22-of-Julia)
An "assert build" of Julia is a build that was built with both `FORCE_ASSERTIONS=1` and `LLVM_ASSERTIONS=1`. To build an assert build, define both of the following variables in your `Make.user` file:
```julia
FORCE_ASSERTIONS=1
LLVM_ASSERTIONS=1
```
Please note that assert builds of Julia will be slower than regular (non-assert) builds.


## [Building 32-bit Julia on a 64-bit machine](#Building-32-bit-Julia-on-a-64-bit-machine)
Occasionally, bugs specific to 32-bit architectures may arise, and when this happens it is useful to be able to debug the problem on your local machine. Since most modern 64-bit systems support running programs built for 32-bit ones, if you don't have to recompile Julia from source (e.g. you mainly need to inspect the behavior of a 32-bit Julia without having to touch the C code), you can likely use a 32-bit build of Julia for your system that you can obtain from the [official downloads page](https://julialang.org/downloads/). However, if you do need to recompile Julia from source one option is to use a Docker container of a 32-bit system. At least for now, building a 32-bit version of Julia is relatively straightforward using [ubuntu 32-bit docker images](https://hub.docker.com/r/i386/ubuntu). In brief, after setting up `docker` here are the required steps:
```sh
$ docker pull i386/ubuntu
$ docker run --platform i386 -i -t i386/ubuntu /bin/bash
```
At this point you should be in a 32-bit machine console (note that `uname` reports the host architecture, so will still say 64-bit, but this will not affect the Julia build). You can add packages and compile code; when you `exit`, all the changes will be lost, so be sure to finish your analysis in a single session or set up a copy/pastable script you can use to set up your environment.
From this point, you should
```sh
# apt update
```


## [Building 32-bit Julia on a 64-bit machine](#Building-32-bit-Julia-on-a-64-bit-machine)
(Note that `sudo` isn't installed, but neither is it necessary since you are running as `root`, so you can omit `sudo` from all commands.)
Then add all the [build dependencies](#required-build-tools-and-external-libraries), a console-based editor of your choice, `git`, and anything else you'll need (e.g., `gdb`, `rr`, etc). Pick a directory to work in and `git clone` Julia, check out the branch you wish to debug, and build Julia as usual.


## [Update the version number of a dependency](#Update-the-version-number-of-a-dependency)
There are two types of builds
1.  Build everything (`deps/` and `src/`) from source code. (Add `USE_BINARYBUILDER=0` to `Make.user`, see [Building Julia](#building-julia))
2.  Build from source (`src/`) with pre-compiled dependencies (default)
When you want to update the version number of a dependency in `deps/`, you may want to use the following checklist:
```md
### Check list

Version numbers:
- [ ] `deps/$(libname).version`: `LIBNAME_VER`, `LIBNAME_BRANCH`, `LIBNAME_SHA1` and `LIBNAME_JLL_VER`
- [ ] `stdlib/$(LIBNAME_JLL_NAME)_jll/Project.toml`: `version`

Checksum:
- [ ] `deps/checksums/$(libname)`
- [ ] `deps/checksums/$(LIBNAME_JLL_NAME)-*/`: `md5` and `sha512`

Patches:
- [ ] `deps/$(libname).mk`
- [ ] `deps/patches/$(libname)-*.patch`
```
Note:
-   For specific dependencies, some items in the checklist may not exist.
-   For checksum file, it may be **a single file** without a suffix, or **a folder** containing two files.


### [Example: OpenLibm](#Example:-OpenLibm)
1.  Update Version numbers in `deps/openlibm.version`
    -   `OPENLIBM_VER := 0.X.Y`
    -   `OPENLIBM_BRANCH = v0.X.Y`
    -   `OPENLIBM_SHA1 = new-sha1-hash`
2.  Update Version number in `stdlib/OpenLibm_jll/Project.toml`
    -   `version = "0.X.Y+0"`
3.  Update checksums in `deps/checksums/openlibm`
    -   `make -f contrib/refresh_checksums.mk openlibm`
4.  Check if the patch files `deps/patches/openlibm-*.patch` exist
    -   if patches don't exist, skip.
    -   if patches exist, check if they have been merged into the new version and need to be removed. When deleting a patch, remember to modify the corresponding Makefile file (`deps/openlibm.mk`).
------------------------------------------------------------------------


# Linux · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/build/linux/


# [Linux](#Linux)
-   GCC version 4.7 or later is required to build Julia.
-   To use external shared libraries not in the system library search path, set `USE_SYSTEM_XXX=1` and `LDFLAGS=-Wl,-rpath,/path/to/dir/contains/libXXX.so` in `Make.user`.
-   Instead of setting `LDFLAGS`, putting the library directory into the environment variable `LD_LIBRARY_PATH` (at both compile and run time) also works.
-   The `USE_SYSTEM_*` flags should be used with caution. These are meant only for troubleshooting, porting, and packaging, where package maintainers work closely with the Julia developers to make sure that Julia is built correctly. Production use cases should use the officially provided binaries. Issues arising from the use of these flags will generally not be accepted.
-   See also the [external dependencies](../build/#required-build-tools-and-external-libraries).


## [Architecture Customization](#Architecture-Customization)
Julia can be built for a non-generic architecture by configuring the `ARCH` Makefile variable in a `Make.user` file. See the appropriate section of `Make.inc` for additional customization options, such as `MARCH` and [`JULIA_CPU_TARGET`](../../../manual/environment-variables/#JULIA_CPU_TARGET).
For example, to build for Pentium 4, set `MARCH=pentium4` and install the necessary system libraries for linking. On Ubuntu, these may include lib32gfortran-6-dev, lib32gcc1, and lib32stdc++6, among others.
You can also set `MARCH=native` in `Make.user` for a maximum-performance build customized for the current machine CPU.


## [Linux Build Troubleshooting](#Linux-Build-Troubleshooting)
|                   Problem |                                                                                Possible Solution |
|--------------------------:|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
|    OpenBLAS build failure | Set one of the following build options in `Make.user` and build again: \
\
 `OPENBLAS_TARGET_ARCH=BARCELONA` (AMD CPUs) or `OPENBLAS_TARGET_ARCH=NEHALEM` (Intel CPUs)\
Set `OPENBLAS_DYNAMIC_ARCH = 0` to disable compiling multiple architectures in a single binary.\
\
\
 `OPENBLAS_NO_AVX2 = 1` disables AVX2 instructions, allowing OpenBLAS to compile with `OPENBLAS_DYNAMIC_ARCH = 1` using old versions of binutils \
\
 `USE_SYSTEM_BLAS=1` uses the system provided `libblas` \
\
Set `LIBBLAS=-lopenblas` and `LIBBLASNAME=libopenblas` to force the use of the system provided OpenBLAS when multiple BLAS versions are installed. \
\
\
\
\
 If you get an error that looks like `` ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1709: Error: no such instruction: `vpermpd $ 0xb1,%ymm0,%ymm0' ``, then you need to set `OPENBLAS_DYNAMIC_ARCH = 0` or `OPENBLAS_NO_AVX2 = 1`, or you need a newer version of `binutils` (2.18 or newer). ([Issue #7653](https://github.com/JuliaLang/julia/issues/7653))\
\
 If the linker cannot find `gfortran` and you get an error like `julia /usr/bin/x86_64-linux-gnu-ld: cannot find -lgfortran`, check the path with `gfortran -print-file-name=libgfortran.so` and use the output to export something similar to this: `export LDFLAGS=-L/usr/lib/gcc/x86_64-linux-gnu/8/`. See [Issue #6150](https://github.com/JuliaLang/julia/issues/6150#issuecomment-37546803).\
 |
| Illegal Instruction error |                                                                                Check if your CPU supports AVX while your OS does not (e.g. through virtualization, as described in [this issue](https://github.com/JuliaLang/julia/issues/3263)). |


## [Linux Build Troubleshooting](#Linux-Build-Troubleshooting)
------------------------------------------------------------------------


# macOS · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/build/macos/


# [macOS](#macOS)
You need to have the current Xcode command line utilities installed: run `xcode-select --install` in the terminal. You will need to rerun this terminal command after each macOS update, otherwise you may run into errors involving missing libraries or headers.
The dependent libraries are now built with [BinaryBuilder](https://binarybuilder.org) and will be automatically downloaded. This is the preferred way to build Julia source. In case you want to build them all on your own, you will need a 64-bit gfortran to compile Julia dependencies.
```bash
brew install gcc
```
If you have set `LD_LIBRARY_PATH` or `DYLD_LIBRARY_PATH` in your `.bashrc` or equivalent, Julia may be unable to find various libraries that come bundled with it. These environment variables need to be unset for Julia to work.
------------------------------------------------------------------------


# Windows · The Julia Language
Source: https://docs.julialang.org/en/v1/devdocs/build/windows/