## [Array and Vectorized Operators and Functions](#man-array-and-vectorized-operators-and-functions)
The following operators are supported for arrays:
1.  Unary arithmetic – `-`, `+`
2.  Binary arithmetic – `-`, `+`, `*`, `/`, `\`, `^`
3.  Comparison – `==`, `!=`, `≈` ([`isapprox`](../../base/math/#Base.isapprox)), `≉`
To enable convenient vectorization of mathematical and other operations, Julia [provides the dot syntax](../functions/#man-vectorized) `f.(args...)`, e.g. `sin.(x)` or `min.(x, y)`, for elementwise operations over arrays or mixtures of arrays and scalars (a [Broadcasting](#Broadcasting) operation); these have the additional advantage of "fusing" into a single loop when combined with other dot calls, e.g. `sin.(cos.(x))`.
Also, *every* binary operator supports a [dot version](../mathematical-operations/#man-dot-operators) that can be applied to arrays (and combinations of arrays and scalars) in such [fused broadcasting operations](../functions/#man-vectorized), e.g. `z .== sin.(x .* y)`.
Note that comparisons such as `==` operate on whole arrays, giving a single boolean answer. Use dot operators like `.==` for elementwise comparisons. (For comparison operations like `<`, *only* the elementwise `.<` version is applicable to arrays.)


## [Array and Vectorized Operators and Functions](#man-array-and-vectorized-operators-and-functions)
Also notice the difference between `max.(a,b)`, which [`broadcast`](../../base/arrays/#Base.Broadcast.broadcast)s [`max`](../../base/math/#Base.max) elementwise over `a` and `b`, and [`maximum(a)`](../../base/collections/#Base.maximum), which finds the largest value within `a`. The same relationship holds for `min.(a, b)` and `minimum(a)`.


## [Broadcasting](#Broadcasting)
It is sometimes useful to perform element-by-element binary operations on arrays of different sizes, such as adding a vector to each column of a matrix. An inefficient way to do this would be to replicate the vector to the size of the matrix:
```julia-repl
julia> a = rand(2, 1); A = rand(2, 3);

julia> repeat(a, 1, 3) + A
2×3 Array{Float64,2}:
 1.20813  1.82068  1.25387
 1.56851  1.86401  1.67846
```
This is wasteful when dimensions get large, so Julia provides [`broadcast`](../../base/arrays/#Base.Broadcast.broadcast), which expands singleton dimensions in array arguments to match the corresponding dimension in the other array without using extra memory, and applies the given function elementwise:
```julia-repl
julia> broadcast(+, a, A)
2×3 Array{Float64,2}:
 1.20813  1.82068  1.25387
 1.56851  1.86401  1.67846

julia> b = rand(1,2)
1×2 Array{Float64,2}:
 0.867535  0.00457906

julia> broadcast(+, a, b)
2×2 Array{Float64,2}:
 1.71056  0.847604
 1.73659  0.873631
```


## [Broadcasting](#Broadcasting)
[Dotted operators](../mathematical-operations/#man-dot-operators) such as `.+` and `.*` are equivalent to `broadcast` calls (except that they fuse, as [described above](#man-array-and-vectorized-operators-and-functions)). There is also a [`broadcast!`](../../base/arrays/#Base.Broadcast.broadcast!) function to specify an explicit destination (which can also be accessed in a fusing fashion by `.=` assignment). In fact, `f.(args...)` is equivalent to `broadcast(f, args...)`, providing a convenient syntax to broadcast any function ([dot syntax](../functions/#man-vectorized)). Nested "dot calls" `f.(...)` (including calls to `.+` etcetera) [automatically fuse](../mathematical-operations/#man-dot-operators) into a single `broadcast` call.
Additionally, [`broadcast`](../../base/arrays/#Base.Broadcast.broadcast) is not limited to arrays (see the function documentation); it also handles scalars, tuples and other collections. By default, only some argument types are considered scalars, including (but not limited to) `Number`s, `String`s, `Symbol`s, `Type`s, `Function`s and some common singletons like `missing` and `nothing`. All other arguments are iterated over or indexed into elementwise.
```julia-repl
julia> convert.(Float32, [1, 2])
2-element Vector{Float32}:
 1.0
 2.0

julia> ceil.(UInt8, [1.2 3.4; 5.6 6.7])
2×2 Matrix{UInt8}:
 0x02  0x04
 0x06  0x07

julia> string.(1:3, ". ", ["First", "Second", "Third"])
3-element Vector{String}:
 "1. First"
 "2. Second"
 "3. Third"
```


## [Broadcasting](#Broadcasting)
Sometimes, you want a container (like an array) that would normally participate in broadcast to be "protected" from broadcast's behavior of iterating over all of its elements. By placing it inside another container (like a single element [`Tuple`](../../base/base/#Core.Tuple)) broadcast will treat it as a single value.
```julia-repl
julia> ([1, 2, 3], [4, 5, 6]) .+ ([1, 2, 3],)
([2, 4, 6], [5, 7, 9])

julia> ([1, 2, 3], [4, 5, 6]) .+ tuple([1, 2, 3])
([2, 4, 6], [5, 7, 9])
```


## [Implementation](#Implementation)
The base array type in Julia is the abstract type [`AbstractArray{T,N}`](../../base/arrays/#Core.AbstractArray). It is parameterized by the number of dimensions `N` and the element type `T`. [`AbstractVector`](../../base/arrays/#Base.AbstractVector) and [`AbstractMatrix`](../../base/arrays/#Base.AbstractMatrix) are aliases for the 1-d and 2-d cases. Operations on `AbstractArray` objects are defined using higher level operators and functions, in a way that is independent of the underlying storage. These operations generally work correctly as a fallback for any specific array implementation.


## [Implementation](#Implementation)
The `AbstractArray` type includes anything vaguely array-like, and implementations of it might be quite different from conventional arrays. For example, elements might be computed on request rather than stored. However, any concrete `AbstractArray{T,N}` type should generally implement at least [`size(A)`](../../base/arrays/#Base.size) (returning an `Int` tuple), [`getindex(A, i)`](../../base/arrays/#Base.getindex-Tuple%7BType,%20Vararg%7BAny%7D%7D) and [`getindex(A, i1, ..., iN)`](../../base/collections/#Base.getindex); mutable arrays should also implement [`setindex!`](../../base/collections/#Base.setindex!). It is recommended that these operations have nearly constant time complexity, as otherwise some array functions may be unexpectedly slow. Concrete types should also typically provide a [`similar(A, T=eltype(A), dims=size(A))`](../../base/arrays/#Base.similar) method, which is used to allocate a similar array for [`copy`](../../base/base/#Base.copy) and other out-of-place operations. No matter how an `AbstractArray{T,N}` is represented internally, `T` is the type of object returned by *integer* indexing (`A[1, ..., 1]`, when `A` is not empty) and `N` should be the length of the tuple returned by [`size`](../../base/arrays/#Base.size). For more details on defining custom `AbstractArray` implementations, see the [array interface guide in the interfaces chapter](../interfaces/#man-interface-array).


## [Implementation](#Implementation)
`DenseArray` is an abstract subtype of `AbstractArray` intended to include all arrays where elements are stored contiguously in column-major order (see [additional notes in Performance Tips](../performance-tips/#man-performance-column-major)). The [`Array`](../../base/arrays/#Core.Array) type is a specific instance of `DenseArray`; [`Vector`](../../base/arrays/#Base.Vector) and [`Matrix`](../../base/arrays/#Base.Matrix) are aliases for the 1-d and 2-d cases. Very few operations are implemented specifically for `Array` beyond those that are required for all `AbstractArray`s; much of the array library is implemented in a generic manner that allows all custom arrays to behave similarly.


## [Implementation](#Implementation)
`SubArray` is a specialization of `AbstractArray` that performs indexing by sharing memory with the original array rather than by copying it. A `SubArray` is created with the [`view`](../../base/arrays/#Base.view) function, which is called the same way as [`getindex`](../../base/collections/#Base.getindex) (with an array and a series of index arguments). The result of [`view`](../../base/arrays/#Base.view) looks the same as the result of [`getindex`](../../base/collections/#Base.getindex), except the data is left in place. [`view`](../../base/arrays/#Base.view) stores the input index vectors in a `SubArray` object, which can later be used to index the original array indirectly. By putting the [`@views`](../../base/arrays/#Base.@views) macro in front of an expression or block of code, any `array[...]` slice in that expression will be converted to create a `SubArray` view instead.
[`BitArray`](../../base/arrays/#Base.BitArray)s are space-efficient "packed" boolean arrays, which store one bit per boolean value. They can be used similarly to `Array{Bool}` arrays (which store one byte per boolean value), and can be converted to/from the latter via `Array(bitarray)` and `BitArray(array)`, respectively.


## [Implementation](#Implementation)
An array is "strided" if it is stored in memory with well-defined spacings (strides) between its elements. A strided array with a supported element type may be passed to an external (non-Julia) library like BLAS or LAPACK by simply passing its [`pointer`](../../base/c/#Base.pointer) and the stride for each dimension. The [`stride(A, d)`](../../base/arrays/#Base.stride) is the distance between elements along dimension `d`. For example, the builtin `Array` returned by `rand(5,7,2)` has its elements arranged contiguously in column major order. This means that the stride of the first dimension — the spacing between elements in the same column — is `1`:
```julia-repl
julia> A = rand(5, 7, 2);

julia> stride(A, 1)
1
```
The stride of the second dimension is the spacing between elements in the same row, skipping as many elements as there are in a single column (`5`). Similarly, jumping between the two "pages" (in the third dimension) requires skipping `5*7 == 35` elements. The [`strides`](../../base/arrays/#Base.strides) of this array is the tuple of these three numbers together:
```julia-repl
julia> strides(A)
(1, 5, 35)
```


## [Implementation](#Implementation)
In this particular case, the number of elements skipped *in memory* matches the number of *linear indices* skipped. This is only the case for contiguous arrays like `Array` (and other `DenseArray` subtypes) and is not true in general. Views with range indices are a good example of *non-contiguous* strided arrays; consider `V = @view A[1:3:4, 2:2:6, 2:-1:1]`. This view `V` refers to the same memory as `A` but is skipping and re-arranging some of its elements. The stride of the first dimension of `V` is `3` because we're only selecting every third row from our original array:
```julia-repl
julia> V = @view A[1:3:4, 2:2:6, 2:-1:1];

julia> stride(V, 1)
3
```
This view is similarly selecting every other column from our original `A` — and thus it needs to skip the equivalent of two five-element columns when moving between indices in the second dimension:
```julia-repl
julia> stride(V, 2)
10
```
The third dimension is interesting because its order is reversed! Thus to get from the first "page" to the second one it must go *backwards* in memory, and so its stride in this dimension is negative!
```julia-repl
julia> stride(V, 3)
-35
```


## [Implementation](#Implementation)
This means that the `pointer` for `V` is actually pointing into the middle of `A`'s memory block, and it refers to elements both backwards and forwards in memory. See the [interface guide for strided arrays](../interfaces/#man-interface-strided-arrays) for more details on defining your own strided arrays. [`StridedVector`](../../base/arrays/#Base.StridedVector) and [`StridedMatrix`](../../base/arrays/#Base.StridedMatrix) are convenient aliases for many of the builtin array types that are considered strided arrays, allowing them to dispatch to select specialized implementations that call highly tuned and optimized BLAS and LAPACK functions using just the pointer and strides.
It is worth emphasizing that strides are about offsets in memory rather than indexing. If you are looking to convert between linear (single-index) indexing and cartesian (multi-index) indexing, see [`LinearIndices`](../../base/arrays/#Base.LinearIndices) and [`CartesianIndices`](../../base/arrays/#Base.IteratorsMD.CartesianIndices).
-   [1](#citeref-1)*iid*, independently and identically distributed.
------------------------------------------------------------------------


# Missing Values · The Julia Language
Source: https://docs.julialang.org/en/v1/manual/missing/


# [Missing Values](#missing)
Julia provides support for representing missing values in the statistical sense. This is for situations where no value is available for a variable in an observation, but a valid value theoretically exists. Missing values are represented via the [`missing`](#missing) object, which is the singleton instance of the type [`Missing`](../../base/base/#Base.Missing). `missing` is equivalent to [`NULL` in SQL](https://en.wikipedia.org/wiki/NULL_(SQL)) and [`NA` in R](https://cran.r-project.org/doc/manuals/r-release/R-lang.html#NA-handling), and behaves like them in most situations.


## [Propagation of Missing Values](#Propagation-of-Missing-Values)
`missing` values *propagate* automatically when passed to standard mathematical operators and functions. For these functions, uncertainty about the value of one of the operands induces uncertainty about the result. In practice, this means a math operation involving a `missing` value generally returns `missing`:
```julia-repl
julia> missing + 1
missing

julia> "a" * missing
missing

julia> abs(missing)
missing
```
Since `missing` is a normal Julia object, this propagation rule only works for functions which have opted in to implement this behavior. This can be achieved by:
-   adding a specific method defined for arguments of type `Missing`,
-   accepting arguments of this type, and passing them to functions which propagate them (like standard math operators).
Packages should consider whether it makes sense to propagate missing values when defining new functions, and define methods appropriately if this is the case. Passing a `missing` value to a function which does not have a method accepting arguments of type `Missing` throws a [`MethodError`](../../base/base/#Core.MethodError), just like for any other type.
Functions that do not propagate `missing` values can be made to do so by wrapping them in the `passmissing` function provided by the [Missings.jl](https://github.com/JuliaData/Missings.jl) package. For example, `f(x)` becomes `passmissing(f)(x)`.


## [Equality and Comparison Operators](#Equality-and-Comparison-Operators)
Standard equality and comparison operators follow the propagation rule presented above: if any of the operands is `missing`, the result is `missing`. Here are a few examples:
```julia-repl
julia> missing == 1
missing

julia> missing == missing
missing

julia> missing < 1
missing

julia> 2 >= missing
missing
```
In particular, note that `missing == missing` returns `missing`, so `==` cannot be used to test whether a value is missing. To test whether `x` is `missing`, use [`ismissing(x)`](../../base/base/#Base.ismissing).
Special comparison operators [`isequal`](../../base/base/#Base.isequal) and [`===`](../../base/base/#Core.:===) are exceptions to the propagation rule. They will always return a `Bool` value, even in the presence of `missing` values, considering `missing` as equal to `missing` and as different from any other value. They can therefore be used to test whether a value is `missing`:
```julia-repl
julia> missing === 1
false

julia> isequal(missing, 1)
false

julia> missing === missing
true

julia> isequal(missing, missing)
true
```
The [`isless`](../../base/base/#Base.isless) operator is another exception: `missing` is considered as greater than any other value. This operator is used by [`sort!`](../../base/sort/#Base.sort!), which therefore places `missing` values after all other values:
```julia-repl
julia> isless(1, missing)
true

julia> isless(missing, Inf)
false

julia> isless(missing, missing)
false
```


## [Logical operators](#Logical-operators)
Logical (or boolean) operators [`|`](../../base/math/#Base.:%7C), [`&`](../../base/math/#Base.:&) and [`xor`](../../base/math/#Base.xor) are another special case since they only propagate `missing` values when it is logically required. For these operators, whether or not the result is uncertain, depends on the particular operation. This follows the well-established rules of [*three-valued logic*](https://en.wikipedia.org/wiki/Three-valued_logic) which are implemented by e.g. `NULL` in SQL and `NA` in R. This abstract definition corresponds to a relatively natural behavior which is best explained via concrete examples.
Let us illustrate this principle with the logical "or" operator [`|`](../../base/math/#Base.:%7C). Following the rules of boolean logic, if one of the operands is `true`, the value of the other operand does not have an influence on the result, which will always be `true`:
```julia-repl
julia> true | true
true

julia> true | false
true

julia> false | true
true
```
Based on this observation, we can conclude if one of the operands is `true` and the other `missing`, we know that the result is `true` in spite of the uncertainty about the actual value of one of the operands. If we had been able to observe the actual value of the second operand, it could only be `true` or `false`, and in both cases the result would be `true`. Therefore, in this particular case, missingness does *not* propagate:
```julia-repl
julia> true | missing
true

julia> missing | true
true
```


## [Logical operators](#Logical-operators)
On the contrary, if one of the operands is `false`, the result could be either `true` or `false` depending on the value of the other operand. Therefore, if that operand is `missing`, the result has to be `missing` too:
```julia-repl
julia> false | true
true

julia> true | false
true

julia> false | false
false

julia> false | missing
missing

julia> missing | false
missing
```
The behavior of the logical "and" operator [`&`](../../base/math/#Base.:&) is similar to that of the `|` operator, with the difference that missingness does not propagate when one of the operands is `false`. For example, when that is the case of the first operand:
```julia-repl
julia> false & false
false

julia> false & true
false

julia> false & missing
false
```
On the other hand, missingness propagates when one of the operands is `true`, for example the first one:
```julia-repl
julia> true & true
true

julia> true & false
false

julia> true & missing
missing
```
Finally, the "exclusive or" logical operator [`xor`](../../base/math/#Base.xor) always propagates `missing` values, since both operands always have an effect on the result. Also note that the negation operator [`!`](../../base/math/#Base.:!) returns `missing` when the operand is `missing`, just like other unary operators.


## [Control Flow and Short-Circuiting Operators](#Control-Flow-and-Short-Circuiting-Operators)
Control flow operators including [`if`](../../base/base/#if), [`while`](../../base/base/#while) and the [ternary operator](../control-flow/#man-conditional-evaluation) `x ? y : z` do not allow for missing values. This is because of the uncertainty about whether the actual value would be `true` or `false` if we could observe it. This implies we do not know how the program should behave. In this case, a [`TypeError`](../../base/base/#Core.TypeError) is thrown as soon as a `missing` value is encountered in this context:
```julia-repl
julia> if missing
           println("here")
       end
ERROR: TypeError: non-boolean (Missing) used in boolean context
```
For the same reason, contrary to logical operators presented above, the short-circuiting boolean operators [`&&`](../../base/math/#&&) and [`||`](../../base/math/#%7C%7C) do not allow for `missing` values in situations where the value of the operand determines whether the next operand is evaluated or not. For example:
```julia-repl
julia> missing || false
ERROR: TypeError: non-boolean (Missing) used in boolean context

julia> missing && false
ERROR: TypeError: non-boolean (Missing) used in boolean context

julia> true && missing && false
ERROR: TypeError: non-boolean (Missing) used in boolean context
```


## [Control Flow and Short-Circuiting Operators](#Control-Flow-and-Short-Circuiting-Operators)
In contrast, there is no error thrown when the result can be determined without the `missing` values. This is the case when the code short-circuits before evaluating the `missing` operand, and when the `missing` operand is the last one:
```julia-repl
julia> true && missing
missing

julia> false && missing
false
```


## [Arrays With Missing Values](#Arrays-With-Missing-Values)
Arrays containing missing values can be created like other arrays:
```julia-repl
julia> [1, missing]
2-element Vector{Union{Missing, Int64}}:
 1
  missing
```
As this example shows, the element type of such arrays is `Union{Missing, T}`, with `T` the type of the non-missing values. This reflects the fact that array entries can be either of type `T` (here, `Int64`) or of type `Missing`. This kind of array uses an efficient memory storage equivalent to an `Array{T}` holding the actual values combined with an `Array{UInt8}` indicating the type of the entry (i.e. whether it is `Missing` or `T`).
Arrays allowing for missing values can be constructed with the standard syntax. Use `Array{Union{Missing, T}}(missing, dims)` to create arrays filled with missing values:
```julia-repl
julia> Array{Union{Missing, String}}(missing, 2, 3)
2×3 Matrix{Union{Missing, String}}:
 missing  missing  missing
 missing  missing  missing
```
Using `undef` or `similar` may currently give an array filled with `missing`, but this is not the correct way to obtain such an array. Use a `missing` constructor as shown above instead.
An array with element type allowing `missing` entries (e.g. `Vector{Union{Missing, T}}`) which does not contain any `missing` entries can be converted to an array type that does not allow for `missing` entries (e.g. `Vector{T}`) using [`convert`](../../base/base/#Base.convert). If the array contains `missing` values, a `MethodError` is thrown during conversion:


## [Arrays With Missing Values](#Arrays-With-Missing-Values)
```julia-repl
julia> x = Union{Missing, String}["a", "b"]
2-element Vector{Union{Missing, String}}:
 "a"
 "b"

julia> convert(Array{String}, x)
2-element Vector{String}:
 "a"
 "b"

julia> y = Union{Missing, String}[missing, "b"]
2-element Vector{Union{Missing, String}}:
 missing
 "b"

julia> convert(Array{String}, y)
ERROR: MethodError: Cannot `convert` an object of type Missing to an object of type String
```


## [Skipping Missing Values](#Skipping-Missing-Values)
Since `missing` values propagate with standard mathematical operators, reduction functions return `missing` when called on arrays which contain missing values:
```julia-repl
julia> sum([1, missing])
missing
```
In this situation, use the [`skipmissing`](../../base/base/#Base.skipmissing) function to skip missing values:
```julia-repl
julia> sum(skipmissing([1, missing]))
1
```
This convenience function returns an iterator which filters out `missing` values efficiently. It can therefore be used with any function which supports iterators:
```julia-repl
julia> x = skipmissing([3, missing, 2, 1])
skipmissing(Union{Missing, Int64}[3, missing, 2, 1])

julia> maximum(x)
3

julia> sum(x)
6

julia> mapreduce(sqrt, +, x)
4.146264369941973
```
Objects created by calling `skipmissing` on an array can be indexed using indices from the parent array. Indices corresponding to missing values are not valid for these objects, and an error is thrown when trying to use them (they are also skipped by `keys` and `eachindex`):
```julia-repl
julia> x[1]
3

julia> x[2]
ERROR: MissingException: the value at index (2,) is missing
[...]
```
This allows functions which operate on indices to work in combination with `skipmissing`. This is notably the case for search and find functions. These functions return indices valid for the object returned by `skipmissing`, and are also the indices of the matching entries *in the parent array*:


## [Skipping Missing Values](#Skipping-Missing-Values)
```julia-repl
julia> findall(==(1), x)
1-element Vector{Int64}:
 4

julia> findfirst(!iszero, x)
1

julia> argmax(x)
1
```
Use [`collect`](../../base/collections/#Base.collect-Tuple%7BAny%7D) to extract non-`missing` values and store them in an array:
```julia-repl
julia> collect(x)
3-element Vector{Int64}:
 3
 2
 1
```


## [Logical Operations on Arrays](#Logical-Operations-on-Arrays)
The three-valued logic described above for logical operators is also used by logical functions applied to arrays. Thus, array equality tests using the [`==`](../../base/math/#Base.:==) operator return `missing` whenever the result cannot be determined without knowing the actual value of the `missing` entry. In practice, this means `missing` is returned if all non-missing values of the compared arrays are equal, but one or both arrays contain missing values (possibly at different positions):
```julia-repl
julia> [1, missing] == [2, missing]
false

julia> [1, missing] == [1, missing]
missing

julia> [1, 2, missing] == [1, missing, 2]
missing
```
As for single values, use [`isequal`](../../base/base/#Base.isequal) to treat `missing` values as equal to other `missing` values, but different from non-missing values:
```julia-repl
julia> isequal([1, missing], [1, missing])
true

julia> isequal([1, 2, missing], [1, missing, 2])
false
```
Functions [`any`](../../base/collections/#Base.any-Tuple%7BAny%7D) and [`all`](../../base/collections/#Base.all-Tuple%7BAny%7D) also follow the rules of three-valued logic. Thus, returning `missing` when the result cannot be determined:
```julia-repl
julia> all([true, missing])
missing

julia> all([false, missing])
false

julia> any([true, missing])
true

julia> any([false, missing])
missing
```
------------------------------------------------------------------------


# Networking and Streams · The Julia Language
Source: https://docs.julialang.org/en/v1/manual/networking-and-streams/


# [Networking and Streams](#Networking-and-Streams)
Julia provides a rich interface to deal with streaming I/O objects such as terminals, pipes and TCP sockets. These objects allow data to be sent and received in a stream-like fashion, which means that data is processed sequentially as it becomes available. This interface, though asynchronous at the system level, is presented in a synchronous manner to the programmer. This is achieved by making heavy use of Julia cooperative threading ([coroutine](../control-flow/#man-tasks)) functionality.


## [Basic Stream I/O](#Basic-Stream-I/O)
All Julia streams expose at least a [`read`](../../base/io-network/#Base.read) and a [`write`](../../base/io-network/#Base.write) method, taking the stream as their first argument, e.g.:
```julia-repl
julia> write(stdout, "Hello World");  # suppress return value 11 with ;
Hello World
julia> read(stdin, Char)

'\n': ASCII/Unicode U+000a (category Cc: Other, control)
```
Note that [`write`](../../base/io-network/#Base.write) returns 11, the number of bytes (in `"Hello World"`) written to [`stdout`](../../base/io-network/#Base.stdout), but this return value is suppressed with the `;`.
Here Enter was pressed again so that Julia would read the newline. Now, as you can see from this example, [`write`](../../base/io-network/#Base.write) takes the data to write as its second argument, while [`read`](../../base/io-network/#Base.read) takes the type of the data to be read as the second argument.
For example, to read a simple byte array, we could do:
```julia-repl
julia> x = zeros(UInt8, 4)
4-element Array{UInt8,1}:
 0x00
 0x00
 0x00
 0x00

julia> read!(stdin, x)
abcd
4-element Array{UInt8,1}:
 0x61
 0x62
 0x63
 0x64
```
However, since this is slightly cumbersome, there are several convenience methods provided. For example, we could have written the above as:
```julia-repl
julia> read(stdin, 4)
abcd
4-element Array{UInt8,1}:
 0x61
 0x62
 0x63
 0x64
```
or if we had wanted to read the entire line instead:
```julia-repl
julia> readline(stdin)
abcd
"abcd"
```


## [Basic Stream I/O](#Basic-Stream-I/O)
Note that depending on your terminal settings, your TTY ("teletype terminal") may be line buffered and might thus require an additional enter before `stdin` data is sent to Julia. When running Julia from the command line in a TTY, output is sent to the console by default, and standard input is read from the keyboard.
To read every line from [`stdin`](../../base/io-network/#Base.stdin) you can use [`eachline`](../../base/io-network/#Base.eachline):
```julia
for line in eachline(stdin)
    print("Found $line")
end
```
or [`read`](../../base/io-network/#Base.read) if you wanted to read by character instead:
```julia
while !eof(stdin)
    x = read(stdin, Char)
    println("Found: $x")
end
```


## [Text I/O](#Text-I/O)
Note that the [`write`](../../base/io-network/#Base.write) method mentioned above operates on binary streams. In particular, values do not get converted to any canonical text representation but are written out as is:
```julia-repl
julia> write(stdout, 0x61);  # suppress return value 1 with ;
a
```
Note that `a` is written to [`stdout`](../../base/io-network/#Base.stdout) by the [`write`](../../base/io-network/#Base.write) function and that the returned value is `1` (since `0x61` is one byte).
For text I/O, use the [`print`](../../base/io-network/#Base.print) or [`show`](../../base/io-network/#Base.show-Tuple%7BIO,%20Any%7D) methods, depending on your needs (see the documentation for these two methods for a detailed discussion of the difference between them):
```julia-repl
julia> print(stdout, 0x61)
97
```
See [Custom pretty-printing](../types/#man-custom-pretty-printing) for more information on how to implement display methods for custom types.


## [IO Output Contextual Properties](#IO-Output-Contextual-Properties)
Sometimes IO output can benefit from the ability to pass contextual information into show methods. The [`IOContext`](../../base/io-network/#Base.IOContext) object provides this framework for associating arbitrary metadata with an IO object. For example, `:compact => true` adds a hinting parameter to the IO object that the invoked show method should print a shorter output (if applicable). See the [`IOContext`](../../base/io-network/#Base.IOContext) documentation for a list of common properties.


## [Working with Files](#Working-with-Files)
You can write content to a file with the `write(filename::String, content)` method:
```julia-repl
julia> write("hello.txt", "Hello, World!")
13
```
*(`13` is the number of bytes written.)*
You can read the contents of a file with the `read(filename::String)` method, or `read(filename::String, String)` to the contents as a string:
```julia-repl
julia> read("hello.txt", String)
"Hello, World!"
```


### [Advanced: streaming files](#Advanced:-streaming-files)
The `read` and `write` methods above allow you to read and write file contents. Like many other environments, Julia also has an [`open`](../../base/io-network/#Base.open) function, which takes a filename and returns an [`IOStream`](../../base/io-network/#Base.IOStream) object that you can use to read and write things from the file. For example, if we have a file, `hello.txt`, whose contents are `Hello, World!`:
```julia-repl
julia> f = open("hello.txt")
IOStream(<file hello.txt>)

julia> readlines(f)
1-element Array{String,1}:
 "Hello, World!"
```
If you want to write to a file, you can open it with the write (`"w"`) flag:
```julia-repl
julia> f = open("hello.txt","w")
IOStream(<file hello.txt>)

julia> write(f,"Hello again.")
12
```
If you examine the contents of `hello.txt` at this point, you will notice that it is empty; nothing has actually been written to disk yet. This is because the `IOStream` must be closed before the write is actually flushed to disk:
```julia-repl
julia> close(f)
```
Examining `hello.txt` again will show its contents have been changed.
Opening a file, doing something to its contents, and closing it again is a very common pattern. To make this easier, there exists another invocation of [`open`](../../base/io-network/#Base.open) which takes a function as its first argument and filename as its second, opens the file, calls the function with the file as an argument, and then closes it again. For example, given a function:


### [Advanced: streaming files](#Advanced:-streaming-files)
```julia
function read_and_capitalize(f::IOStream)
    return uppercase(read(f, String))
end
```
You can call:
```julia-repl
julia> open(read_and_capitalize, "hello.txt")
"HELLO AGAIN."
```
to open `hello.txt`, call `read_and_capitalize` on it, close `hello.txt` and return the capitalized contents.
To avoid even having to define a named function, you can use the `do` syntax, which creates an anonymous function on the fly:
```julia-repl
julia> open("hello.txt") do f
           uppercase(read(f, String))
       end
"HELLO AGAIN."
```
If you want to redirect stdout to a file
    out_file = open("output.txt", "w")
    # Redirect stdout to file
    redirect_stdout(out_file) do
        # Your code here
        println("This output goes to `out_file` via the `stdout` variable.")
    end
    # Close file
    close(out_file)
Redirecting stdout to a file can help you save and analyze program output, automate processes, and meet compliance requirements.


## [A simple TCP example](#A-simple-TCP-example)
Let's jump right in with a simple example involving TCP sockets. This functionality is in a standard library package called `Sockets`. Let's first create a simple server:
```julia-repl
julia> using Sockets

julia> errormonitor(@async begin
           server = listen(2000)
           while true
               sock = accept(server)
               println("Hello World\n")
           end
       end)
Task (runnable) @0x00007fd31dc11ae0
```
To those familiar with the Unix socket API, the method names will feel familiar, though their usage is somewhat simpler than the raw Unix socket API. The first call to [`listen`](../../stdlib/Sockets/#Sockets.listen-Tuple%7BAny%7D) will create a server waiting for incoming connections on the specified port (2000) in this case. The same function may also be used to create various other kinds of servers:
```julia-repl
julia> listen(2000) # Listens on localhost:2000 (IPv4)
Sockets.TCPServer(active)

julia> listen(ip"127.0.0.1",2000) # Equivalent to the first
Sockets.TCPServer(active)

julia> listen(ip"::1",2000) # Listens on localhost:2000 (IPv6)
Sockets.TCPServer(active)

julia> listen(IPv4(0),2001) # Listens on port 2001 on all IPv4 interfaces
Sockets.TCPServer(active)

julia> listen(IPv6(0),2001) # Listens on port 2001 on all IPv6 interfaces
Sockets.TCPServer(active)

julia> listen("testsocket") # Listens on a UNIX domain socket
Sockets.PipeServer(active)

julia> listen("\\\\.\\pipe\\testsocket") # Listens on a Windows named pipe
Sockets.PipeServer(active)
```


## [A simple TCP example](#A-simple-TCP-example)
Note that the return type of the last invocation is different. This is because this server does not listen on TCP, but rather on a named pipe (Windows) or UNIX domain socket. Also note that Windows named pipe format has to be a specific pattern such that the name prefix (`\\.\pipe\`) uniquely identifies the [file type](https://docs.microsoft.com/windows/desktop/ipc/pipe-names). The difference between TCP and named pipes or UNIX domain sockets is subtle and has to do with the [`accept`](../../stdlib/Sockets/#Sockets.accept) and [`connect`](../../stdlib/Distributed/#Sockets.connect-Tuple%7BClusterManager,%20Int64,%20WorkerConfig%7D) methods. The [`accept`](../../stdlib/Sockets/#Sockets.accept) method retrieves a connection to the client that is connecting on the server we just created, while the [`connect`](../../stdlib/Distributed/#Sockets.connect-Tuple%7BClusterManager,%20Int64,%20WorkerConfig%7D) function connects to a server using the specified method. The [`connect`](../../stdlib/Distributed/#Sockets.connect-Tuple%7BClusterManager,%20Int64,%20WorkerConfig%7D) function takes the same arguments as [`listen`](../../stdlib/Sockets/#Sockets.listen-Tuple%7BAny%7D), so, assuming the environment (i.e. host, cwd, etc.) is the same you should be able to pass the same arguments to [`connect`](../../stdlib/Distributed/#Sockets.connect-Tuple%7BClusterManager,%20Int64,%20WorkerConfig%7D) as you did to listen to establish the connection. So let's try that out (after having created the server above):


## [A simple TCP example](#A-simple-TCP-example)
```julia-repl
julia> connect(2000)
TCPSocket(open, 0 bytes waiting)

julia> Hello World
```
As expected we saw "Hello World" printed. So, let's actually analyze what happened behind the scenes. When we called [`connect`](../../stdlib/Distributed/#Sockets.connect-Tuple%7BClusterManager,%20Int64,%20WorkerConfig%7D), we connect to the server we had just created. Meanwhile, the accept function returns a server-side connection to the newly created socket and prints "Hello World" to indicate that the connection was successful.
A great strength of Julia is that since the API is exposed synchronously even though the I/O is actually happening asynchronously, we didn't have to worry about callbacks or even making sure that the server gets to run. When we called [`connect`](../../stdlib/Distributed/#Sockets.connect-Tuple%7BClusterManager,%20Int64,%20WorkerConfig%7D) the current task waited for the connection to be established and only continued executing after that was done. In this pause, the server task resumed execution (because a connection request was now available), accepted the connection, printed the message and waited for the next client. Reading and writing works in the same way. To see this, consider the following simple echo server:


## [A simple TCP example](#A-simple-TCP-example)
```julia-repl
julia> errormonitor(@async begin
           server = listen(2001)
           while true
               sock = accept(server)
               @async while isopen(sock)
                   write(sock, readline(sock, keep=true))
               end
           end
       end)
Task (runnable) @0x00007fd31dc12e60

julia> clientside = connect(2001)
TCPSocket(RawFD(28) open, 0 bytes waiting)

julia> errormonitor(@async while isopen(clientside)
           write(stdout, readline(clientside, keep=true))
       end)
Task (runnable) @0x00007fd31dc11870

julia> println(clientside,"Hello World from the Echo Server")
Hello World from the Echo Server
```
As with other streams, use [`close`](../../base/io-network/#Base.close) to disconnect the socket:
```julia-repl
julia> close(clientside)
```


## [Resolving IP Addresses](#Resolving-IP-Addresses)
One of the [`connect`](../../stdlib/Distributed/#Sockets.connect-Tuple%7BClusterManager,%20Int64,%20WorkerConfig%7D) methods that does not follow the [`listen`](../../stdlib/Sockets/#Sockets.listen-Tuple%7BAny%7D) methods is `connect(host::String,port)`, which will attempt to connect to the host given by the `host` parameter on the port given by the `port` parameter. It allows you to do things like:
```julia-repl
julia> connect("google.com", 80)
TCPSocket(RawFD(30) open, 0 bytes waiting)
```
At the base of this functionality is [`getaddrinfo`](../../stdlib/Sockets/#Sockets.getaddrinfo), which will do the appropriate address resolution:
```julia-repl
julia> getaddrinfo("google.com")
ip"74.125.226.225"
```


## [Asynchronous I/O](#Asynchronous-I/O)
All I/O operations exposed by [`Base.read`](../../base/io-network/#Base.read) and [`Base.write`](../../base/io-network/#Base.write) can be performed asynchronously through the use of [coroutines](../control-flow/#man-tasks). You can create a new coroutine to read from or write to a stream using the [`@async`](../../base/parallel/#Base.@async) macro:
```julia-repl
julia> task = @async open("foo.txt", "w") do io
           write(io, "Hello, World!")
       end;

julia> wait(task)

julia> readlines("foo.txt")
1-element Array{String,1}:
 "Hello, World!"
```
It's common to run into situations where you want to perform multiple asynchronous operations concurrently and wait until they've all completed. You can use the [`@sync`](../../base/parallel/#Base.@sync) macro to cause your program to block until all of the coroutines it wraps around have exited:
```julia-repl
julia> using Sockets

julia> @sync for hostname in ("google.com", "github.com", "julialang.org")
           @async begin
               conn = connect(hostname, 80)
               write(conn, "GET / HTTP/1.1\r\nHost:$(hostname)\r\n\r\n")
               readline(conn, keep=true)
               println("Finished connection to $(hostname)")
           end
       end
Finished connection to google.com
Finished connection to julialang.org
Finished connection to github.com
```


## [Multicast](#Multicast)
Julia supports [multicast](https://datatracker.ietf.org/doc/html/rfc1112) over IPv4 and IPv6 using the User Datagram Protocol ([UDP](https://datatracker.ietf.org/doc/html/rfc768)) as transport.
Unlike the Transmission Control Protocol ([TCP](https://datatracker.ietf.org/doc/html/rfc793)), UDP makes almost no assumptions about the needs of the application. TCP provides flow control (it accelerates and decelerates to maximize throughput), reliability (lost or corrupt packets are automatically retransmitted), sequencing (packets are ordered by the operating system before they are given to the application), segment size, and session setup and teardown. UDP provides no such features.
A common use for UDP is in multicast applications. TCP is a stateful protocol for communication between exactly two devices. UDP can use special multicast addresses to allow simultaneous communication between many devices.


### [Receiving IP Multicast Packets](#Receiving-IP-Multicast-Packets)
To transmit data over UDP multicast, simply `recv` on the socket, and the first packet received will be returned. Note that it may not be the first packet that you sent however!
```julia
using Sockets
group = ip"228.5.6.7"
socket = Sockets.UDPSocket()
bind(socket, ip"0.0.0.0", 6789)
join_multicast_group(socket, group)
println(String(recv(socket)))
leave_multicast_group(socket, group)
close(socket)
```


### [Sending IP Multicast Packets](#Sending-IP-Multicast-Packets)
To transmit data over UDP multicast, simply `send` to the socket. Notice that it is not necessary for a sender to join the multicast group.
```julia
using Sockets
group = ip"228.5.6.7"
socket = Sockets.UDPSocket()
send(socket, group, 6789, "Hello over IPv4")
close(socket)
```


### [IPv6 Example](#IPv6-Example)
This example gives the same functionality as the previous program, but uses IPv6 as the network-layer protocol.
Listener:
```julia
using Sockets
group = Sockets.IPv6("ff05::5:6:7")
socket = Sockets.UDPSocket()
bind(socket, Sockets.IPv6("::"), 6789)
join_multicast_group(socket, group)
println(String(recv(socket)))
leave_multicast_group(socket, group)
close(socket)
```
Sender:
```julia
using Sockets
group = Sockets.IPv6("ff05::5:6:7")
socket = Sockets.UDPSocket()
send(socket, group, 6789, "Hello over IPv6")
close(socket)
```
------------------------------------------------------------------------


# Parallel Computing · The Julia Language
Source: https://docs.julialang.org/en/v1/manual/parallel-computing/


# [Parallel Computing](#Parallel-Computing)
Julia supports these four categories of concurrent and parallel programming:
1.  **Asynchronous "tasks", or coroutines**:
    Julia Tasks allow suspending and resuming computations for I/O, event handling, producer-consumer processes, and similar patterns. Tasks can synchronize through operations like [`wait`](../../base/parallel/#Base.wait) and [`fetch`](../../base/parallel/#Base.fetch-Tuple%7BTask%7D), and communicate via [`Channel`](../../base/parallel/#Base.Channel)s. While strictly not parallel computing by themselves, Julia lets you schedule [`Task`](../../base/parallel/#Core.Task)s on several threads.
2.  **Multi-threading**:
    Julia's [multi-threading](../multi-threading/#man-multithreading) provides the ability to schedule Tasks simultaneously on more than one thread or CPU core, sharing memory. This is usually the easiest way to get parallelism on one's PC or on a single large multi-core server. Julia's multi-threading is composable. When one multi-threaded function calls another multi-threaded function, Julia will schedule all the threads globally on available resources, without oversubscribing.
3.  **Distributed computing**:


# [Parallel Computing](#Parallel-Computing)
    Distributed computing runs multiple Julia processes with separate memory spaces. These can be on the same computer or multiple computers. The [`Distributed`](../../stdlib/Distributed/#man-distributed) standard library provides the capability for remote execution of a Julia function. With this basic building block, it is possible to build many different kinds of distributed computing abstractions. Packages like [`DistributedArrays.jl`](https://github.com/JuliaParallel/DistributedArrays.jl) are an example of such an abstraction. On the other hand, packages like [`MPI.jl`](https://github.com/JuliaParallel/MPI.jl) and [`Elemental.jl`](https://github.com/JuliaParallel/Elemental.jl) provide access to the existing MPI ecosystem of libraries.
4.  **GPU computing**:
    The Julia GPU compiler provides the ability to run Julia code natively on GPUs. There is a rich ecosystem of Julia packages that target GPUs. The [JuliaGPU.org](https://juliagpu.org) website provides a list of capabilities, supported GPUs, related packages and documentation.
------------------------------------------------------------------------


# Asynchronous Programming · The Julia Language
Source: https://docs.julialang.org/en/v1/manual/asynchronous-programming/


# [Asynchronous Programming](#man-asynchronous)
When a program needs to interact with the outside world, for example communicating with another machine over the internet, operations in the program may need to happen in an unpredictable order. Say your program needs to download a file. We would like to initiate the download operation, perform other operations while we wait for it to complete, and then resume the code that needs the downloaded file when it is available. This sort of scenario falls in the domain of asynchronous programming, sometimes also referred to as concurrent programming (since, conceptually, multiple things are happening at once).


# [Asynchronous Programming](#man-asynchronous)
To address these scenarios, Julia provides [`Task`](../../base/parallel/#Core.Task)s (also known by several other names, such as symmetric coroutines, lightweight threads, cooperative multitasking, or one-shot continuations). When a piece of computing work (in practice, executing a particular function) is designated as a [`Task`](../../base/parallel/#Core.Task), it becomes possible to interrupt it by switching to another [`Task`](../../base/parallel/#Core.Task). The original [`Task`](../../base/parallel/#Core.Task) can later be resumed, at which point it will pick up right where it left off. At first, this may seem similar to a function call. However there are two key differences. First, switching tasks does not use any space, so any number of task switches can occur without consuming the call stack. Second, switching among tasks can occur in any order, unlike function calls, where the called function must finish executing before control returns to the calling function.


## [Basic Task operations](#Basic-Task-operations)
You can think of a `Task` as a handle to a unit of computational work to be performed. It has a create-start-run-finish lifecycle. Tasks are created by calling the `Task` constructor on a 0-argument function to run, or using the [`@task`](../../base/parallel/#Base.@task) macro:
```julia-repl
julia> t = @task begin; sleep(5); println("done"); end
Task (runnable) @0x00007f13a40c0eb0
```
`@task x` is equivalent to `Task(()->x)`.
This task will wait for five seconds, and then print `done`. However, it has not started running yet. We can run it whenever we're ready by calling [`schedule`](../../base/parallel/#Base.schedule):
```julia-repl
julia> schedule(t);
```
If you try this in the REPL, you will see that `schedule` returns immediately. That is because it simply adds `t` to an internal queue of tasks to run. Then, the REPL will print the next prompt and wait for more input. Waiting for keyboard input provides an opportunity for other tasks to run, so at that point `t` will start. `t` calls [`sleep`](../../base/parallel/#Base.sleep), which sets a timer and stops execution. If other tasks have been scheduled, they could run then. After five seconds, the timer fires and restarts `t`, and you will see `done` printed. `t` is then finished.
The [`wait`](../../base/parallel/#Base.wait) function blocks the calling task until some other task finishes. So for example if you type
```julia-repl
julia> schedule(t); wait(t)
```


## [Basic Task operations](#Basic-Task-operations)
instead of only calling `schedule`, you will see a five second pause before the next input prompt appears. That is because the REPL is waiting for `t` to finish before proceeding.
It is common to want to create a task and schedule it right away, so the macro [`@async`](../../base/parallel/#Base.@async) is provided for that purpose –- `@async x` is equivalent to `schedule(@task x)`.


## [Communicating with Channels](#Communicating-with-Channels)
In some problems, the various pieces of required work are not naturally related by function calls; there is no obvious "caller" or "callee" among the jobs that need to be done. An example is the producer-consumer problem, where one complex procedure is generating values and another complex procedure is consuming them. The consumer cannot simply call a producer function to get a value, because the producer may have more values to generate and so might not yet be ready to return. With tasks, the producer and consumer can both run as long as they need to, passing values back and forth as necessary.
Julia provides a [`Channel`](../../base/parallel/#Base.Channel) mechanism for solving this problem. A [`Channel`](../../base/parallel/#Base.Channel) is a waitable first-in first-out queue which can have multiple tasks reading from and writing to it.
Let's define a producer task, which produces values via the [`put!`](../../base/parallel/#Base.put!-Tuple%7BChannel,%20Any%7D) call. To consume values, we need to schedule the producer to run in a new task. A special [`Channel`](../../base/parallel/#Base.Channel) constructor which accepts a 1-arg function as an argument can be used to run a task bound to a channel. We can then [`take!`](../../base/io-network/#Base.take!-Tuple%7BBase.GenericIOBuffer%7D) values repeatedly from the channel object:


## [Communicating with Channels](#Communicating-with-Channels)
```julia-repl
julia> function producer(c::Channel)
           put!(c, "start")
           for n=1:4
               put!(c, 2n)
           end
           put!(c, "stop")
       end;

julia> chnl = Channel(producer);

julia> take!(chnl)
"start"

julia> take!(chnl)
2

julia> take!(chnl)
4

julia> take!(chnl)
6

julia> take!(chnl)
8

julia> take!(chnl)
"stop"
```
One way to think of this behavior is that `producer` was able to return multiple times. Between calls to [`put!`](../../base/parallel/#Base.put!-Tuple%7BChannel,%20Any%7D), the producer's execution is suspended and the consumer has control.
The returned [`Channel`](../../base/parallel/#Base.Channel) can be used as an iterable object in a `for` loop, in which case the loop variable takes on all the produced values. The loop is terminated when the channel is closed.
```julia-repl
julia> for x in Channel(producer)
           println(x)
       end
start
2
4
6
8
stop
```
Note that we did not have to explicitly close the channel in the producer. This is because the act of binding a [`Channel`](../../base/parallel/#Base.Channel) to a [`Task`](../../base/parallel/#Core.Task) associates the open lifetime of a channel with that of the bound task. The channel object is closed automatically when the task terminates. Multiple channels can be bound to a task, and vice-versa.


## [Communicating with Channels](#Communicating-with-Channels)
While the [`Task`](../../base/parallel/#Core.Task) constructor expects a 0-argument function, the [`Channel`](../../base/parallel/#Base.Channel) method that creates a task-bound channel expects a function that accepts a single argument of type [`Channel`](../../base/parallel/#Base.Channel). A common pattern is for the producer to be parameterized, in which case a partial function application is needed to create a 0 or 1 argument [anonymous function](../functions/#man-anonymous-functions).
For [`Task`](../../base/parallel/#Core.Task) objects this can be done either directly or by use of a convenience macro:
```julia
function mytask(myarg)
    ...
end

taskHdl = Task(() -> mytask(7))
# or, equivalently
taskHdl = @task mytask(7)
```julia
To orchestrate more advanced work distribution patterns, [`bind`](../../stdlib/Sockets/#Base.bind) and [`schedule`](../../base/parallel/#Base.schedule) can be used in conjunction with [`Task`](../../base/parallel/#Core.Task) and [`Channel`](../../base/parallel/#Base.Channel) constructors to explicitly link a set of channels with a set of producer/consumer tasks.


### [More on Channels](#More-on-Channels)
A channel can be visualized as a pipe, i.e., it has a write end and a read end :
-   Multiple writers in different tasks can write to the same channel concurrently via [`put!`](../../base/parallel/#Base.put!-Tuple%7BChannel,%20Any%7D) calls.
-   Multiple readers in different tasks can read data concurrently via [`take!`](../../base/io-network/#Base.take!-Tuple%7BBase.GenericIOBuffer%7D) calls.
-   As an example:
    ```julia
    # Given Channels c1 and c2,
    c1 = Channel(32)
    c2 = Channel(32)

    # and a function `foo` which reads items from c1, processes the item read
    # and writes a result to c2,
    function foo()
        while true
            data = take!(c1)
            [...]               # process data
            put!(c2, result)    # write out result
        end
    end

    # we can schedule `n` instances of `foo` to be active concurrently.
    for _ in 1:n
        errormonitor(@async foo())
    end
    ```
-   Channels are created via the `Channel{T}(sz)` constructor. The channel will only hold objects of type `T`. If the type is not specified, the channel can hold objects of any type. `sz` refers to the maximum number of elements that can be held in the channel at any time. For example, `Channel(32)` creates a channel that can hold a maximum of 32 objects of any type. A `Channel{MyType}(64)` can hold up to 64 objects of `MyType` at any time.


### [More on Channels](#More-on-Channels)
-   If a [`Channel`](../../base/parallel/#Base.Channel) is empty, readers (on a [`take!`](../../base/io-network/#Base.take!-Tuple%7BBase.GenericIOBuffer%7D) call) will block until data is available.
-   If a [`Channel`](../../base/parallel/#Base.Channel) is full, writers (on a [`put!`](../../base/parallel/#Base.put!-Tuple%7BChannel,%20Any%7D) call) will block until space becomes available.
-   [`isready`](../../base/parallel/#Base.isready-Tuple%7BChannel%7D) tests for the presence of any object in the channel, while [`wait`](../../base/parallel/#Base.wait) waits for an object to become available.
-   A [`Channel`](../../base/parallel/#Base.Channel) is in an open state initially. This means that it can be read from and written to freely via [`take!`](../../base/io-network/#Base.take!-Tuple%7BBase.GenericIOBuffer%7D) and [`put!`](../../base/parallel/#Base.put!-Tuple%7BChannel,%20Any%7D) calls. [`close`](../../base/io-network/#Base.close) closes a [`Channel`](../../base/parallel/#Base.Channel). On a closed [`Channel`](../../base/parallel/#Base.Channel), [`put!`](../../base/parallel/#Base.put!-Tuple%7BChannel,%20Any%7D) will fail. For example:
    ```julia-repl
    julia> c = Channel(2);

    julia> put!(c, 1) # `put!` on an open channel succeeds
    1

    julia> close(c);

    julia> put!(c, 2) # `put!` on a closed channel throws an exception.
    ERROR: InvalidStateException: Channel is closed.
    Stacktrace:
    [...]
    ```


### [More on Channels](#More-on-Channels)
-   [`take!`](../../base/io-network/#Base.take!-Tuple%7BBase.GenericIOBuffer%7D) and [`fetch`](../../base/parallel/#Base.fetch-Tuple%7BTask%7D) (which retrieves but does not remove the value) on a closed channel successfully return any existing values until it is emptied. Continuing the above example:
    ```julia-repl
    julia> fetch(c) # Any number of `fetch` calls succeed.
    1

    julia> fetch(c)
    1

    julia> take!(c) # The first `take!` removes the value.
    1

    julia> take!(c) # No more data available on a closed channel.
    ERROR: InvalidStateException: Channel is closed.
    Stacktrace:
    [...]
    ```
Consider a simple example using channels for inter-task communication. We start 4 tasks to process data from a single `jobs` channel. Jobs, identified by an id (`job_id`), are written to the channel. Each task in this simulation reads a `job_id`, waits for a random amount of time and writes back a tuple of `job_id` and the simulated time to the results channel. Finally all the `results` are printed out.


### [More on Channels](#More-on-Channels)
```julia-repl
julia> const jobs = Channel{Int}(32);

julia> const results = Channel{Tuple}(32);

julia> function do_work()
           for job_id in jobs
               exec_time = rand()
               sleep(exec_time)                # simulates elapsed time doing actual work
                                               # typically performed externally.
               put!(results, (job_id, exec_time))
           end
       end;

julia> function make_jobs(n)
           for i in 1:n
               put!(jobs, i)
           end
       end;

julia> n = 12;

julia> errormonitor(@async make_jobs(n)); # feed the jobs channel with "n" jobs

julia> for i in 1:4 # start 4 tasks to process requests in parallel
           errormonitor(@async do_work())
       end

julia> @elapsed while n > 0 # print out results
           job_id, exec_time = take!(results)
           println("$job_id finished in $(round(exec_time; digits=2)) seconds")
           global n = n - 1
       end
4 finished in 0.22 seconds
3 finished in 0.45 seconds
1 finished in 0.5 seconds
7 finished in 0.14 seconds
2 finished in 0.78 seconds
5 finished in 0.9 seconds
9 finished in 0.36 seconds
6 finished in 0.87 seconds
8 finished in 0.79 seconds
10 finished in 0.64 seconds
12 finished in 0.5 seconds
11 finished in 0.97 seconds
0.029772311
```
Instead of `errormonitor(t)`, a more robust solution may be to use `bind(results, t)`, as that will not only log any unexpected failures, but also force the associated resources to close and propagate the exception everywhere.


## [More task operations](#More-task-operations)
Task operations are built on a low-level primitive called [`yieldto`](../../base/parallel/#Base.yieldto). `yieldto(task, value)` suspends the current task, switches to the specified `task`, and causes that task's last [`yieldto`](../../base/parallel/#Base.yieldto) call to return the specified `value`. Notice that [`yieldto`](../../base/parallel/#Base.yieldto) is the only operation required to use task-style control flow; instead of calling and returning we are always just switching to a different task. This is why this feature is also called "symmetric coroutines"; each task is switched to and from using the same mechanism.
[`yieldto`](../../base/parallel/#Base.yieldto) is powerful, but most uses of tasks do not invoke it directly. Consider why this might be. If you switch away from the current task, you will probably want to switch back to it at some point, but knowing when to switch back, and knowing which task has the responsibility of switching back, can require considerable coordination. For example, [`put!`](../../base/parallel/#Base.put!-Tuple%7BChannel,%20Any%7D) and [`take!`](../../base/io-network/#Base.take!-Tuple%7BBase.GenericIOBuffer%7D) are blocking operations, which, when used in the context of channels maintain state to remember who the consumers are. Not needing to manually keep track of the consuming task is what makes [`put!`](../../base/parallel/#Base.put!-Tuple%7BChannel,%20Any%7D) easier to use than the low-level [`yieldto`](../../base/parallel/#Base.yieldto).


## [More task operations](#More-task-operations)
In addition to [`yieldto`](../../base/parallel/#Base.yieldto), a few other basic functions are needed to use tasks effectively.
-   [`current_task`](../../base/parallel/#Base.current_task) gets a reference to the currently-running task.
-   [`istaskdone`](../../base/parallel/#Base.istaskdone) queries whether a task has exited.
-   [`istaskstarted`](../../base/parallel/#Base.istaskstarted) queries whether a task has run yet.
-   [`task_local_storage`](../../base/parallel/#Base.task_local_storage-Tuple%7BAny%7D) manipulates a key-value store specific to the current task.


## [Tasks and events](#Tasks-and-events)
Most task switches occur as a result of waiting for events such as I/O requests, and are performed by a scheduler included in Julia Base. The scheduler maintains a queue of runnable tasks, and executes an event loop that restarts tasks based on external events such as message arrival.
The basic function for waiting for an event is [`wait`](../../base/parallel/#Base.wait). Several objects implement [`wait`](../../base/parallel/#Base.wait); for example, given a `Process` object, [`wait`](../../base/parallel/#Base.wait) will wait for it to exit. [`wait`](../../base/parallel/#Base.wait) is often implicit; for example, a [`wait`](../../base/parallel/#Base.wait) can happen inside a call to [`read`](../../base/io-network/#Base.read) to wait for data to be available.
In all of these cases, [`wait`](../../base/parallel/#Base.wait) ultimately operates on a [`Condition`](../../base/parallel/#Base.Condition) object, which is in charge of queueing and restarting tasks. When a task calls [`wait`](../../base/parallel/#Base.wait) on a [`Condition`](../../base/parallel/#Base.Condition), the task is marked as non-runnable, added to the condition's queue, and switches to the scheduler. The scheduler will then pick another task to run, or block waiting for external events. If all goes well, eventually an event handler will call [`notify`](../../base/parallel/#Base.notify) on the condition, which causes tasks waiting for that condition to become runnable again.


## [Tasks and events](#Tasks-and-events)
A task created explicitly by calling [`Task`](../../base/parallel/#Core.Task) is initially not known to the scheduler. This allows you to manage tasks manually using [`yieldto`](../../base/parallel/#Base.yieldto) if you wish. However, when such a task waits for an event, it still gets restarted automatically when the event happens, as you would expect.
------------------------------------------------------------------------


# Multi-Threading · The Julia Language
Source: https://docs.julialang.org/en/v1/manual/multi-threading/


# [Multi-Threading](#man-multithreading)
Visit this [blog post](https://julialang.org/blog/2019/07/multithreading/) for a presentation of Julia multi-threading features.


## [Starting Julia with multiple threads](#Starting-Julia-with-multiple-threads)
By default, Julia starts up with a single thread of execution. This can be verified by using the command [`Threads.nthreads()`](../../base/multi-threading/#Base.Threads.nthreads):
```julia-repl
julia> Threads.nthreads()
1
```
The number of execution threads is controlled either by using the `-t`/`--threads` command line argument or by using the [`JULIA_NUM_THREADS`](../environment-variables/#JULIA_NUM_THREADS) environment variable. When both are specified, then `-t`/`--threads` takes precedence.
The number of threads can either be specified as an integer (`--threads=4`) or as `auto` (`--threads=auto`), where `auto` tries to infer a useful default number of threads to use (see [Command-line Options](../command-line-interface/#command-line-interface) for more details).
The `-t`/`--threads` command line argument requires at least Julia 1.5. In older versions you must use the environment variable instead.
Using `auto` as value of the environment variable [`JULIA_NUM_THREADS`](../environment-variables/#JULIA_NUM_THREADS) requires at least Julia 1.7. In older versions, this value is ignored.
Lets start Julia with 4 threads:
```bash
$ julia --threads 4
```
Let's verify there are 4 threads at our disposal.
```julia-repl
julia> Threads.nthreads()
4
```
But we are currently on the master thread. To check, we use the function [`Threads.threadid`](../../base/multi-threading/#Base.Threads.threadid)
```julia-repl
julia> Threads.threadid()
1
```


## [Starting Julia with multiple threads](#Starting-Julia-with-multiple-threads)
If you prefer to use the environment variable you can set it as follows in Bash (Linux/macOS):
```bash
export JULIA_NUM_THREADS=4
```
C shell on Linux/macOS, CMD on Windows:
```bash
set JULIA_NUM_THREADS=4
```
Powershell on Windows:
```powershell
$env:JULIA_NUM_THREADS=4
```
Note that this must be done *before* starting Julia.
The number of threads specified with `-t`/`--threads` is propagated to worker processes that are spawned using the `-p`/`--procs` or `--machine-file` command line options. For example, `julia -p2 -t2` spawns 1 main process with 2 worker processes, and all three processes have 2 threads enabled. For more fine grained control over worker threads use [`addprocs`](../../stdlib/Distributed/#Distributed.addprocs) and pass `-t`/`--threads` as `exeflags`.


### [Multiple GC Threads](#Multiple-GC-Threads)
The Garbage Collector (GC) can use multiple threads. The amount used is either half the number of compute worker threads or configured by either the `--gcthreads` command line argument or by using the [`JULIA_NUM_GC_THREADS`](../environment-variables/#JULIA_NUM_GC_THREADS) environment variable.
The `--gcthreads` command line argument requires at least Julia 1.10.


## [Threadpools](#man-threadpools)
When a program's threads are busy with many tasks to run, tasks may experience delays which may negatively affect the responsiveness and interactivity of the program. To address this, you can specify that a task is interactive when you [`Threads.@spawn`](../../base/multi-threading/#Base.Threads.@spawn) it:
```julia
using Base.Threads
@spawn :interactive f()
```
Interactive tasks should avoid performing high latency operations, and if they are long duration tasks, should yield frequently.
Julia may be started with one or more threads reserved to run interactive tasks:
```bash
$ julia --threads 3,1
```
The environment variable [`JULIA_NUM_THREADS`](../environment-variables/#JULIA_NUM_THREADS) can also be used similarly:
```bash
export JULIA_NUM_THREADS=3,1
```
This starts Julia with 3 threads in the `:default` threadpool and 1 thread in the `:interactive` threadpool:
```julia-repl
julia> using Base.Threads

julia> nthreadpools()
2

julia> threadpool() # the main thread is in the interactive thread pool
:interactive

julia> nthreads(:default)
3

julia> nthreads(:interactive)
1

julia> nthreads()
3
```
The zero-argument version of `nthreads` returns the number of threads in the default pool.
Depending on whether Julia has been started with interactive threads, the main thread is either in the default or interactive thread pool.
Either or both numbers can be replaced with the word `auto`, which causes Julia to choose a reasonable default.


## [The @threads Macro](#The-@threads-Macro)
Let's work a simple example using our native threads. Let us create an array of zeros:
```julia-repl
julia> a = zeros(10)
10-element Vector{Float64}:
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
 0.0
```
Let us operate on this array simultaneously using 4 threads. We'll have each thread write its thread ID into each location.
Julia supports parallel loops using the [`Threads.@threads`](../../base/multi-threading/#Base.Threads.@threads) macro. This macro is affixed in front of a `for` loop to indicate to Julia that the loop is a multi-threaded region:
```julia-repl
julia> Threads.@threads for i = 1:10
           a[i] = Threads.threadid()
       end
```
The iteration space is split among the threads, after which each thread writes its thread ID to its assigned locations:
```julia-repl
julia> a
10-element Vector{Float64}:
 1.0
 1.0
 1.0
 2.0
 2.0
 2.0
 3.0
 3.0
 4.0
 4.0
```
Note that [`Threads.@threads`](../../base/multi-threading/#Base.Threads.@threads) does not have an optional reduction parameter like [`@distributed`](../../stdlib/Distributed/#Distributed.@distributed).


### [Using @threads without data-races](#Using-@threads-without-data-races)
The concept of a data-race is elaborated on in ["Communication and data races between threads"](#man-communication-and-data-races). For now, just known that a data race can result in incorrect results and dangerous errors.
Lets say we want to make the function `sum_single` below multithreaded.
```julia-repl
julia> function sum_single(a)
           s = 0
           for i in a
               s += i
           end
           s
       end
sum_single (generic function with 1 method)

julia> sum_single(1:1_000_000)
500000500000
```
Simply adding `@threads` exposes a data race with multiple threads reading and writing `s` at the same time.
```julia-repl
julia> function sum_multi_bad(a)
           s = 0
           Threads.@threads for i in a
               s += i
           end
           s
       end
sum_multi_bad (generic function with 1 method)

julia> sum_multi_bad(1:1_000_000)
70140554652
```
Note that the result is not `500000500000` as it should be, and will most likely change each evaluation.
To fix this, buffers that are specific to the task may be used to segment the sum into chunks that are race-free. Here `sum_single` is reused, with its own internal buffer `s`. The input vector `a` is split into `nthreads()` chunks for parallel work. We then use `Threads.@spawn` to create tasks that individually sum each chunk. Finally, we sum the results from each task using `sum_single` again:


### [Using @threads without data-races](#Using-@threads-without-data-races)
```julia-repl
julia> function sum_multi_good(a)
           chunks = Iterators.partition(a, length(a) ÷ Threads.nthreads())
           tasks = map(chunks) do chunk
               Threads.@spawn sum_single(chunk)
           end
           chunk_sums = fetch.(tasks)
           return sum_single(chunk_sums)
       end
sum_multi_good (generic function with 1 method)

julia> sum_multi_good(1:1_000_000)
500000500000
```
Buffers should not be managed based on `threadid()` i.e. `buffers = zeros(Threads.nthreads())` because concurrent tasks can yield, meaning multiple concurrent tasks may use the same buffer on a given thread, introducing risk of data races. Further, when more than one thread is available tasks may change thread at yield points, which is known as [task migration](#man-task-migration).
Another option is the use of atomic operations on variables shared across tasks/threads, which may be more performant depending on the characteristics of the operations.


## [Communication and data-races between threads](#man-communication-and-data-races)
Although Julia's threads can communicate through shared memory, it is notoriously difficult to write correct and data-race free multi-threaded code. Julia's [`Channel`](../../base/parallel/#Base.Channel)s are thread-safe and may be used to communicate safely. There are also sections below that explain how to use [locks](#man-using-locks) and [atomics](#man-atomic-operations) to avoid data-races.


### [Data-race freedom](#Data-race-freedom)
You are entirely responsible for ensuring that your program is data-race free, and nothing promised here can be assumed if you do not observe that requirement. The observed results may be highly unintuitive.
If data-races are introduced, Julia is not memory safe. **Be very careful about reading *any* data if another thread might write to it, as it could result in segmentation faults or worse**. Below are a couple of unsafe ways to access global variables from different threads:
```julia
Thread 1:
global b = false
global a = rand()
global b = true

Thread 2:
while !b; end
bad_read1(a) # it is NOT safe to access `a` here!

Thread 3:
while !@isdefined(a); end
bad_read2(a) # it is NOT safe to access `a` here
```


### [Using locks to avoid data-races](#man-using-locks)
An important tool to avoid data-races, and thereby write thread-safe code, is the concept of a "lock". A lock can be locked and unlocked. If a thread has locked a lock, and not unlocked it, it is said to "hold" the lock. If there is only one lock, and we write code the requires holding the lock to access some data, we can ensure that multiple threads will never access the same data simultaneously. Note that the link between a lock and a variable is made by the programmer, and not the program.
For example, we can create a lock `my_lock`, and lock it while we mutate a variable `my_variable`. This is done most simply with the `@lock` macro:
```julia-repl
julia> my_lock = ReentrantLock();

julia> my_variable = [1, 2, 3];

julia> @lock my_lock my_variable[1] = 100
100
```
By using a similar pattern with the same lock and variable, but on another thread, the operations are free from data-races.
We could have performed the operation above with the functional version of `lock`, in the following two ways:
```julia-repl
julia> lock(my_lock) do
           my_variable[1] = 100
       end
100

julia> begin
           lock(my_lock)
           try
               my_variable[1] = 100
           finally
               unlock(my_lock)
           end
       end
100
```


### [Using locks to avoid data-races](#man-using-locks)
All three options are equivalent. Note how the final version requires an explicit `try`-block to ensure that the lock is always unlocked, whereas the first two version do this internally. One should always use the lock pattern above when changing data (such as assigning to a global or closure variable) accessed by other threads. Failing to do this could have unforeseen and serious consequences.


### [Atomic Operations](#man-atomic-operations)
Julia supports accessing and modifying values *atomically*, that is, in a thread-safe way to avoid [race conditions](https://en.wikipedia.org/wiki/Race_condition). A value (which must be of a primitive type) can be wrapped as [`Threads.Atomic`](../../base/multi-threading/#Base.Threads.Atomic) to indicate it must be accessed in this way. Here we can see an example:
```julia-repl
julia> i = Threads.Atomic{Int}(0);

julia> ids = zeros(4);

julia> old_is = zeros(4);

julia> Threads.@threads for id in 1:4
           old_is[id] = Threads.atomic_add!(i, id)
           ids[id] = id
       end

julia> old_is
4-element Vector{Float64}:
 0.0
 1.0
 7.0
 3.0

julia> i[]
 10

julia> ids
4-element Vector{Float64}:
 1.0
 2.0
 3.0
 4.0
```
Had we tried to do the addition without the atomic tag, we might have gotten the wrong answer due to a race condition. An example of what would happen if we didn't avoid the race:
```julia-repl
julia> using Base.Threads

julia> Threads.nthreads()
4

julia> acc = Ref(0)
Base.RefValue{Int64}(0)

julia> @threads for i in 1:1000
          acc[] += 1
       end

julia> acc[]
926

julia> acc = Atomic{Int64}(0)
Atomic{Int64}(0)

julia> @threads for i in 1:1000
          atomic_add!(acc, 1)
       end

julia> acc[]
1000
```


#### [Per-field atomics](#man-atomics)
We can also use atomics on a more granular level using the [`@atomic`](../../base/multi-threading/#Base.@atomic), [`@atomicswap`](../../base/multi-threading/#Base.@atomicswap), [`@atomicreplace`](../../base/multi-threading/#Base.@atomicreplace) macros, and [`@atomiconce`](../../base/multi-threading/#Base.@atomiconce) macros.
Specific details of the memory model and other details of the design are written in the [Julia Atomics Manifesto](https://gist.github.com/vtjnash/11b0031f2e2a66c9c24d33e810b34ec0), which will later be published formally.
Any field in a struct declaration can be decorated with `@atomic`, and then any write must be marked with `@atomic` also, and must use one of the defined atomic orderings (`:monotonic`, `:acquire`, `:release`, `:acquire_release`, or `:sequentially_consistent`). Any read of an atomic field can also be annotated with an atomic ordering constraint, or will be done with monotonic (relaxed) ordering if unspecified.
Per-field atomics requires at least Julia 1.7.


## [Side effects and mutable function arguments](#Side-effects-and-mutable-function-arguments)
When using multi-threading we have to be careful when using functions that are not [pure](https://en.wikipedia.org/wiki/Pure_function) as we might get a wrong answer. For instance functions that have a [name ending with `!`](../style-guide/#bang-convention) by convention modify their arguments and thus are not pure.


## [@threadcall](#@threadcall)
External libraries, such as those called via [`ccall`](../../base/c/#ccall), pose a problem for Julia's task-based I/O mechanism. If a C library performs a blocking operation, that prevents the Julia scheduler from executing any other tasks until the call returns. (Exceptions are calls into custom C code that call back into Julia, which may then yield, or C code that calls `jl_yield()`, the C equivalent of [`yield`](../../base/parallel/#Base.yield).)
The [`@threadcall`](#@threadcall) macro provides a way to avoid stalling execution in such a scenario. It schedules a C function for execution in a separate thread. A threadpool with a default size of 4 is used for this. The size of the threadpool is controlled via environment variable `UV_THREADPOOL_SIZE`. While waiting for a free thread, and during function execution once a thread is available, the requesting task (on the main Julia event loop) yields to other tasks. Note that `@threadcall` does not return until the execution is complete. From a user point of view, it is therefore a blocking call like other Julia APIs.
It is very important that the called function does not call back into Julia, as it will segfault.
`@threadcall` may be removed/changed in future versions of Julia.


## [Caveats](#Caveats)
At this time, most operations in the Julia runtime and standard libraries can be used in a thread-safe manner, if the user code is data-race free. However, in some areas work on stabilizing thread support is ongoing. Multi-threaded programming has many inherent difficulties, and if a program using threads exhibits unusual or undesirable behavior (e.g. crashes or mysterious results), thread interactions should typically be suspected first.
There are a few specific limitations and warnings to be aware of when using threads in Julia:
-   Base collection types require manual locking if used simultaneously by multiple threads where at least one thread modifies the collection (common examples include `push!` on arrays, or inserting items into a `Dict`).
-   The schedule used by [`@spawn`](../../base/multi-threading/#Base.Threads.@spawn) is nondeterministic and should not be relied on.
-   Compute-bound, non-memory-allocating tasks can prevent garbage collection from running in other threads that are allocating memory. In these cases it may be necessary to insert a manual call to `GC.safepoint()` to allow GC to run. This limitation will be removed in the future.
-   Avoid running top-level operations, e.g. `include`, or `eval` of type, method, and module definitions in parallel.


## [Caveats](#Caveats)
-   Be aware that finalizers registered by a library may break if threads are enabled. This may require some transitional work across the ecosystem before threading can be widely adopted with confidence. See the section on [the safe use of finalizers](#man-finalizers) for further details.


## [Task Migration](#man-task-migration)
After a task starts running on a certain thread it may move to a different thread if the task yields.
Such tasks may have been started with [`@spawn`](../../base/multi-threading/#Base.Threads.@spawn) or [`@threads`](../../base/multi-threading/#Base.Threads.@threads), although the `:static` schedule option for `@threads` does freeze the threadid.
This means that in most cases [`threadid()`](../../base/multi-threading/#Base.Threads.threadid) should not be treated as constant within a task, and therefore should not be used to index into a vector of buffers or stateful objects.
Task migration was introduced in Julia 1.7. Before this tasks always remained on the same thread that they were started on.


## [Safe use of Finalizers](#man-finalizers)
Because finalizers can interrupt any code, they must be very careful in how they interact with any global state. Unfortunately, the main reason that finalizers are used is to update global state (a pure function is generally rather pointless as a finalizer). This leads us to a bit of a conundrum. There are a few approaches to dealing with this problem:
1.  When single-threaded, code could call the internal `jl_gc_enable_finalizers` C function to prevent finalizers from being scheduled inside a critical region. Internally, this is used inside some functions (such as our C locks) to prevent recursion when doing certain operations (incremental package loading, codegen, etc.). The combination of a lock and this flag can be used to make finalizers safe.
2.  A second strategy, employed by Base in a couple places, is to explicitly delay a finalizer until it may be able to acquire its lock non-recursively. The following example demonstrates how this strategy could be applied to `Distributed.finalize_ref`:


## [Safe use of Finalizers](#man-finalizers)
    ```julia
    function finalize_ref(r::AbstractRemoteRef)
        if r.where > 0 # Check if the finalizer is already run
            if islocked(client_refs) || !trylock(client_refs)
                # delay finalizer for later if we aren't free to acquire the lock
                finalizer(finalize_ref, r)
                return nothing
            end
            try # `lock` should always be followed by `try`
                if r.where > 0 # Must check again here
                    # Do actual cleanup here
                    r.where = 0
                end
            finally
                unlock(client_refs)
            end
        end
        nothing
    end
    ```
3.  A related third strategy is to use a yield-free queue. We don't currently have a lock-free queue implemented in Base, but `Base.IntrusiveLinkedListSynchronized{T}` is suitable. This can frequently be a good strategy to use for code with event loops. For example, this strategy is employed by `Gtk.jl` to manage lifetime ref-counting. In this approach, we don't do any explicit work inside the `finalizer`, and instead add it to a queue to run at a safer time. In fact, Julia's task scheduler already uses this, so defining the finalizer as `x -> @spawn do_cleanup(x)` is one example of this approach. Note however that this doesn't control which thread `do_cleanup` runs on, so `do_cleanup` would still need to acquire a lock. That doesn't need to be true if you implement your own queue, as you can explicitly only drain that queue from your thread.


## [Safe use of Finalizers](#man-finalizers)
------------------------------------------------------------------------


# Multi-processing and Distributed Computing · The Julia Language
Source: https://docs.julialang.org/en/v1/manual/distributed-computing/


# [Multi-processing and Distributed Computing](#Multi-processing-and-Distributed-Computing)
An implementation of distributed memory parallel computing is provided by module [`Distributed`](../../stdlib/Distributed/#man-distributed) as part of the standard library shipped with Julia.
Most modern computers possess more than one CPU, and several computers can be combined together in a cluster. Harnessing the power of these multiple CPUs allows many computations to be completed more quickly. There are two major factors that influence performance: the speed of the CPUs themselves, and the speed of their access to memory. In a cluster, it's fairly obvious that a given CPU will have fastest access to the RAM within the same computer (node). Perhaps more surprisingly, similar issues are relevant on a typical multicore laptop, due to differences in the speed of main memory and the [cache](https://www.akkadia.org/drepper/cpumemory.pdf). Consequently, a good multiprocessing environment should allow control over the "ownership" of a chunk of memory by a particular CPU. Julia provides a multiprocessing environment based on message passing to allow programs to run on multiple processes in separate memory domains at once.


# [Multi-processing and Distributed Computing](#Multi-processing-and-Distributed-Computing)
Julia's implementation of message passing is different from other environments such as MPI<sup>[[1\]](#footnote-1)</sup>. Communication in Julia is generally "one-sided", meaning that the programmer needs to explicitly manage only one process in a two-process operation. Furthermore, these operations typically do not look like "message send" and "message receive" but rather resemble higher-level operations like calls to user functions.
Distributed programming in Julia is built on two primitives: *remote references* and *remote calls*. A remote reference is an object that can be used from any process to refer to an object stored on a particular process. A remote call is a request by one process to call a certain function on certain arguments on another (possibly the same) process.
Remote references come in two flavors: [`Future`](../../stdlib/Distributed/#Distributed.Future) and [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel).
A remote call returns a [`Future`](../../stdlib/Distributed/#Distributed.Future) to its result. Remote calls return immediately; the process that made the call proceeds to its next operation while the remote call happens somewhere else. You can wait for a remote call to finish by calling [`wait`](../../base/parallel/#Base.wait) on the returned [`Future`](../../stdlib/Distributed/#Distributed.Future), and you can obtain the full value of the result using [`fetch`](../../base/parallel/#Base.fetch-Tuple%7BTask%7D).


# [Multi-processing and Distributed Computing](#Multi-processing-and-Distributed-Computing)
On the other hand, [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel) s are rewritable. For example, multiple processes can coordinate their processing by referencing the same remote `Channel`.
Each process has an associated identifier. The process providing the interactive Julia prompt always has an `id` equal to 1. The processes used by default for parallel operations are referred to as "workers". When there is only one process, process 1 is considered a worker. Otherwise, workers are considered to be all processes other than process 1. As a result, adding 2 or more processes is required to gain benefits from parallel processing methods like [`pmap`](../../stdlib/Distributed/#Distributed.pmap). Adding a single process is beneficial if you just wish to do other things in the main process while a long computation is running on the worker.
Let's try this out. Starting with `julia -p n` provides `n` worker processes on the local machine. Generally it makes sense for `n` to equal the number of CPU threads (logical cores) on the machine. Note that the `-p` argument implicitly loads module [`Distributed`](../../stdlib/Distributed/#man-distributed).
```julia
$ julia -p 2

julia> r = remotecall(rand, 2, 2, 2)
Future(2, 1, 4, nothing)

julia> s = @spawnat 2 1 .+ fetch(r)
Future(2, 1, 5, nothing)

julia> fetch(s)
2×2 Array{Float64,2}:
 1.18526  1.50912
 1.16296  1.60607
```


# [Multi-processing and Distributed Computing](#Multi-processing-and-Distributed-Computing)
The first argument to [`remotecall`](../../stdlib/Distributed/#Distributed.remotecall-Tuple%7BAny,%20Integer,%20Vararg%7BAny%7D%7D) is the function to call. Most parallel programming in Julia does not reference specific processes or the number of processes available, but [`remotecall`](../../stdlib/Distributed/#Distributed.remotecall-Tuple%7BAny,%20Integer,%20Vararg%7BAny%7D%7D) is considered a low-level interface providing finer control. The second argument to [`remotecall`](../../stdlib/Distributed/#Distributed.remotecall-Tuple%7BAny,%20Integer,%20Vararg%7BAny%7D%7D) is the `id` of the process that will do the work, and the remaining arguments will be passed to the function being called.
As you can see, in the first line we asked process 2 to construct a 2-by-2 random matrix, and in the second line we asked it to add 1 to it. The result of both calculations is available in the two futures, `r` and `s`. The [`@spawnat`](../../stdlib/Distributed/#Distributed.@spawnat) macro evaluates the expression in the second argument on the process specified by the first argument.
Occasionally you might want a remotely-computed value immediately. This typically happens when you read from a remote object to obtain data needed by the next local operation. The function [`remotecall_fetch`](../../stdlib/Distributed/#Distributed.remotecall_fetch-Tuple%7BAny,%20Integer,%20Vararg%7BAny%7D%7D) exists for this purpose. It is equivalent to `fetch(remotecall(...))` but is more efficient.


# [Multi-processing and Distributed Computing](#Multi-processing-and-Distributed-Computing)
```julia-repl
julia> remotecall_fetch(r-> fetch(r)[1, 1], 2, r)
0.18526337335308085
```
This fetches the array on worker 2 and returns the first value. Note, that `fetch` doesn't move any data in this case, since it's executed on the worker that owns the array. One can also write:
```julia-repl
julia> remotecall_fetch(getindex, 2, r, 1, 1)
0.10824216411304866
```
Remember that [`getindex(r,1,1)`](../../base/arrays/#Base.getindex-Tuple%7BType,%20Vararg%7BAny%7D%7D) is [equivalent](../arrays/#man-array-indexing) to `r[1,1]`, so this call fetches the first element of the future `r`.
To make things easier, the symbol `:any` can be passed to [`@spawnat`](../../stdlib/Distributed/#Distributed.@spawnat), which picks where to do the operation for you:
```julia-repl
julia> r = @spawnat :any rand(2,2)
Future(2, 1, 4, nothing)

julia> s = @spawnat :any 1 .+ fetch(r)
Future(3, 1, 5, nothing)

julia> fetch(s)
2×2 Array{Float64,2}:
 1.38854  1.9098
 1.20939  1.57158
```
Note that we used `1 .+ fetch(r)` instead of `1 .+ r`. This is because we do not know where the code will run, so in general a [`fetch`](../../base/parallel/#Base.fetch-Tuple%7BTask%7D) might be required to move `r` to the process doing the addition. In this case, [`@spawnat`](../../stdlib/Distributed/#Distributed.@spawnat) is smart enough to perform the computation on the process that owns `r`, so the [`fetch`](../../base/parallel/#Base.fetch-Tuple%7BTask%7D) will be a no-op (no work is done).


# [Multi-processing and Distributed Computing](#Multi-processing-and-Distributed-Computing)
(It is worth noting that [`@spawnat`](../../stdlib/Distributed/#Distributed.@spawnat) is not built-in but defined in Julia as a [macro](../metaprogramming/#man-macros). It is possible to define your own such constructs.)
An important thing to remember is that, once fetched, a [`Future`](../../stdlib/Distributed/#Distributed.Future) will cache its value locally. Further [`fetch`](../../base/parallel/#Base.fetch-Tuple%7BTask%7D) calls do not entail a network hop. Once all referencing [`Future`](../../stdlib/Distributed/#Distributed.Future)s have fetched, the remote stored value is deleted.


# [Multi-processing and Distributed Computing](#Multi-processing-and-Distributed-Computing)
[`@async`](../../base/parallel/#Base.@async) is similar to [`@spawnat`](../../stdlib/Distributed/#Distributed.@spawnat), but only runs tasks on the local process. We use it to create a "feeder" task for each process. Each task picks the next index that needs to be computed, then waits for its process to finish, then repeats until we run out of indices. Note that the feeder tasks do not begin to execute until the main task reaches the end of the [`@sync`](../../base/parallel/#Base.@sync) block, at which point it surrenders control and waits for all the local tasks to complete before returning from the function. As for v0.7 and beyond, the feeder tasks are able to share state via `nextidx` because they all run on the same process. Even if `Tasks` are scheduled cooperatively, locking may still be required in some contexts, as in [asynchronous I/O](../faq/#faq-async-io). This means context switches only occur at well-defined points: in this case, when [`remotecall_fetch`](../../stdlib/Distributed/#Distributed.remotecall_fetch-Tuple%7BAny,%20Integer,%20Vararg%7BAny%7D%7D) is called. This is the current state of implementation and it may change for future Julia versions, as it is intended to make it possible to run up to N `Tasks` on M `Process`, aka [M:N Threading](https://en.wikipedia.org/wiki/Thread_(computing)#Models). Then a lock acquiring\\releasing model for `nextidx` will be needed, as it is not safe to let multiple processes read-write a resource at the same time.


## [Code Availability and Loading Packages](#code-availability)
Your code must be available on any process that runs it. For example, type the following into the Julia prompt:
```julia-repl
julia> function rand2(dims...)
           return 2*rand(dims...)
       end

julia> rand2(2,2)
2×2 Array{Float64,2}:
 0.153756  0.368514
 1.15119   0.918912

julia> fetch(@spawnat :any rand2(2,2))
ERROR: RemoteException(2, CapturedException(UndefVarError(Symbol("#rand2"))))
Stacktrace:
[...]
```
Process 1 knew about the function `rand2`, but process 2 did not.
Most commonly you'll be loading code from files or packages, and you have a considerable amount of flexibility in controlling which processes load code. Consider a file, `DummyModule.jl`, containing the following code:
```julia
module DummyModule

export MyType, f

mutable struct MyType
    a::Int
end

f(x) = x^2+1

println("loaded")

end
```
In order to refer to `MyType` across all processes, `DummyModule.jl` needs to be loaded on every process. Calling `include("DummyModule.jl")` loads it only on a single process. To load it on every process, use the [`@everywhere`](../../stdlib/Distributed/#Distributed.@everywhere) macro (starting Julia with `julia -p 2`):
```julia-repl
julia> @everywhere include("DummyModule.jl")
loaded
      From worker 3:    loaded
      From worker 2:    loaded
```


## [Code Availability and Loading Packages](#code-availability)
As usual, this does not bring `DummyModule` into scope on any of the process, which requires [`using`](../../base/base/#using) or [`import`](../../base/base/#import). Moreover, when `DummyModule` is brought into scope on one process, it is not on any other:
```julia-repl
julia> using .DummyModule

julia> MyType(7)
MyType(7)

julia> fetch(@spawnat 2 MyType(7))
ERROR: On worker 2:
UndefVarError: `MyType` not defined in `Main`
⋮

julia> fetch(@spawnat 2 DummyModule.MyType(7))
MyType(7)
```
However, it's still possible, for instance, to send a `MyType` to a process which has loaded `DummyModule` even if it's not in scope:
```julia-repl
julia> put!(RemoteChannel(2), MyType(7))
RemoteChannel{Channel{Any}}(2, 1, 13)
```
A file can also be preloaded on multiple processes at startup with the `-L` flag, and a driver script can be used to drive the computation:
```julia
julia -p <n> -L file1.jl -L file2.jl driver.jl
```
The Julia process running the driver script in the example above has an `id` equal to 1, just like a process providing an interactive prompt.
Finally, if `DummyModule.jl` is not a standalone file but a package, then `using DummyModule` will *load* `DummyModule.jl` on all processes, but only bring it into scope on the process where [`using`](../../base/base/#using) was called.


## [Starting and managing worker processes](#Starting-and-managing-worker-processes)
The base Julia installation has in-built support for two types of clusters:
-   A local cluster specified with the `-p` option as shown above.
-   A cluster spanning machines using the `--machine-file` option. This uses a passwordless `ssh` login to start Julia worker processes (from the same path as the current host) on the specified machines. Each machine definition takes the form `[count*][user@]host[:port] [bind_addr[:port]]`. `user` defaults to current user, `port` to the standard ssh port. `count` is the number of workers to spawn on the node, and defaults to 1. The optional `bind-to bind_addr[:port]` specifies the IP address and port that other workers should use to connect to this worker.
While Julia generally strives for backward compatibility, distribution of code to worker processes relies on [`Serialization.serialize`](../../stdlib/Serialization/#Serialization.serialize). As pointed out in the corresponding documentation, this can not be guaranteed to work across different Julia versions, so it is advised that all workers on all machines use the same version.
Functions [`addprocs`](../../stdlib/Distributed/#Distributed.addprocs), [`rmprocs`](../../stdlib/Distributed/#Distributed.rmprocs), [`workers`](../../stdlib/Distributed/#Distributed.workers), and others are available as a programmatic means of adding, removing and querying the processes in a cluster.
```julia-repl
julia> using Distributed

julia> addprocs(2)
2-element Array{Int64,1}:
 2
 3
```


## [Starting and managing worker processes](#Starting-and-managing-worker-processes)
Module [`Distributed`](../../stdlib/Distributed/#man-distributed) must be explicitly loaded on the master process before invoking [`addprocs`](../../stdlib/Distributed/#Distributed.addprocs). It is automatically made available on the worker processes.
Note that workers do not run a `~/.julia/config/startup.jl` startup script, nor do they synchronize their global state (such as command-line switches, global variables, new method definitions, and loaded modules) with any of the other running processes. You may use `addprocs(exeflags="--project")` to initialize a worker with a particular environment, and then `@everywhere using <modulename>` or `@everywhere include("file.jl")`.
Other types of clusters can be supported by writing your own custom `ClusterManager`, as described below in the [ClusterManagers](#ClusterManagers) section.