## [Data Movement](#Data-Movement)
Sending messages and moving data constitute most of the overhead in a distributed program. Reducing the number of messages and the amount of data sent is critical to achieving performance and scalability. To this end, it is important to understand the data movement performed by Julia's various distributed programming constructs.
[`fetch`](../../base/parallel/#Base.fetch-Tuple%7BTask%7D) can be considered an explicit data movement operation, since it directly asks that an object be moved to the local machine. [`@spawnat`](../../stdlib/Distributed/#Distributed.@spawnat) (and a few related constructs) also moves data, but this is not as obvious, hence it can be called an implicit data movement operation. Consider these two approaches to constructing and squaring a random matrix:
Method 1:
```julia-repl
julia> A = rand(1000,1000);

julia> Bref = @spawnat :any A^2;

[...]

julia> fetch(Bref);
```
Method 2:
```julia-repl
julia> Bref = @spawnat :any rand(1000,1000)^2;

[...]

julia> fetch(Bref);
```
The difference seems trivial, but in fact is quite significant due to the behavior of [`@spawnat`](../../stdlib/Distributed/#Distributed.@spawnat). In the first method, a random matrix is constructed locally, then sent to another process where it is squared. In the second method, a random matrix is both constructed and squared on another process. Therefore the second method sends much less data than the first.


## [Data Movement](#Data-Movement)
In this toy example, the two methods are easy to distinguish and choose from. However, in a real program designing data movement might require more thought and likely some measurement. For example, if the first process needs matrix `A` then the first method might be better. Or, if computing `A` is expensive and only the current process has it, then moving it to another process might be unavoidable. Or, if the current process has very little to do between the [`@spawnat`](../../stdlib/Distributed/#Distributed.@spawnat) and `fetch(Bref)`, it might be better to eliminate the parallelism altogether. Or imagine `rand(1000,1000)` is replaced with a more expensive operation. Then it might make sense to add another [`@spawnat`](../../stdlib/Distributed/#Distributed.@spawnat) statement just for this step.


## [Global variables](#Global-variables)
Expressions executed remotely via [`@spawnat`](../../stdlib/Distributed/#Distributed.@spawnat), or closures specified for remote execution using [`remotecall`](../../stdlib/Distributed/#Distributed.remotecall-Tuple%7BAny,%20Integer,%20Vararg%7BAny%7D%7D) may refer to global variables. Global bindings under module `Main` are treated a little differently compared to global bindings in other modules. Consider the following code snippet:
```julia-repl
A = rand(10,10)
remotecall_fetch(()->sum(A), 2)
```
In this case [`sum`](../../base/collections/#Base.sum) MUST be defined in the remote process. Note that `A` is a global variable defined in the local workspace. Worker 2 does not have a variable called `A` under `Main`. The act of shipping the closure `()->sum(A)` to worker 2 results in `Main.A` being defined on 2. `Main.A` continues to exist on worker 2 even after the call [`remotecall_fetch`](../../stdlib/Distributed/#Distributed.remotecall_fetch-Tuple%7BAny,%20Integer,%20Vararg%7BAny%7D%7D) returns. Remote calls with embedded global references (under `Main` module only) manage globals as follows:
-   New global bindings are created on destination workers if they are referenced as part of a remote call.
-   Global constants are declared as constants on remote nodes too.
-   Globals are re-sent to a destination worker only in the context of a remote call, and then only if its value has changed. Also, the cluster does not synchronize global bindings across nodes. For example:


## [Global variables](#Global-variables)
    ```julia
    A = rand(10,10)
    remotecall_fetch(()->sum(A), 2) # worker 2
    A = rand(10,10)
    remotecall_fetch(()->sum(A), 3) # worker 3
    A = nothing
    ```
    Executing the above snippet results in `Main.A` on worker 2 having a different value from `Main.A` on worker 3, while the value of `Main.A` on node 1 is set to `nothing`.
As you may have realized, while memory associated with globals may be collected when they are reassigned on the master, no such action is taken on the workers as the bindings continue to be valid. [`clear!`](../../stdlib/Distributed/#Distributed.clear!) can be used to manually reassign specific globals on remote nodes to `nothing` once they are no longer required. This will release any memory associated with them as part of a regular garbage collection cycle.
Thus programs should be careful referencing globals in remote calls. In fact, it is preferable to avoid them altogether if possible. If you must reference globals, consider using `let` blocks to localize global variables.
For example:
```julia-repl
julia> A = rand(10,10);

julia> remotecall_fetch(()->A, 2);

julia> B = rand(10,10);

julia> let B = B
           remotecall_fetch(()->B, 2)
       end;

julia> @fetchfrom 2 InteractiveUtils.varinfo()
name           size summary
––––––––– ––––––––– ––––––––––––––––––––––
A         800 bytes 10×10 Array{Float64,2}
Base                Module
Core                Module
Main                Module
```


## [Global variables](#Global-variables)
As can be seen, global variable `A` is defined on worker 2, but `B` is captured as a local variable and hence a binding for `B` does not exist on worker 2.


## [Parallel Map and Loops](#Parallel-Map-and-Loops)
Fortunately, many useful parallel computations do not require data movement. A common example is a Monte Carlo simulation, where multiple processes can handle independent simulation trials simultaneously. We can use [`@spawnat`](../../stdlib/Distributed/#Distributed.@spawnat) to flip coins on two processes. First, write the following function in `count_heads.jl`:
```julia
function count_heads(n)
    c::Int = 0
    for i = 1:n
        c += rand(Bool)
    end
    c
end
```
The function `count_heads` simply adds together `n` random bits. Here is how we can perform some trials on two machines, and add together the results:
```julia-repl
julia> @everywhere include_string(Main, $(read("count_heads.jl", String)), "count_heads.jl")

julia> a = @spawnat :any count_heads(100000000)
Future(2, 1, 6, nothing)

julia> b = @spawnat :any count_heads(100000000)
Future(3, 1, 7, nothing)

julia> fetch(a)+fetch(b)
100001564
```


## [Parallel Map and Loops](#Parallel-Map-and-Loops)
This example demonstrates a powerful and often-used parallel programming pattern. Many iterations run independently over several processes, and then their results are combined using some function. The combination process is called a *reduction*, since it is generally tensor-rank-reducing: a vector of numbers is reduced to a single number, or a matrix is reduced to a single row or column, etc. In code, this typically looks like the pattern `x = f(x,v[i])`, where `x` is the accumulator, `f` is the reduction function, and the `v[i]` are the elements being reduced. It is desirable for `f` to be associative, so that it does not matter what order the operations are performed in.
Notice that our use of this pattern with `count_heads` can be generalized. We used two explicit [`@spawnat`](../../stdlib/Distributed/#Distributed.@spawnat) statements, which limits the parallelism to two processes. To run on any number of processes, we can use a *parallel for loop*, running in distributed memory, which can be written in Julia using [`@distributed`](../../stdlib/Distributed/#Distributed.@distributed) like this:
```julia
nheads = @distributed (+) for i = 1:200000000
    Int(rand(Bool))
end
```
This construct implements the pattern of assigning iterations to multiple processes, and combining them with a specified reduction (in this case `(+)`). The result of each iteration is taken as the value of the last expression inside the loop. The whole parallel loop expression itself evaluates to the final answer.


## [Parallel Map and Loops](#Parallel-Map-and-Loops)
Note that although parallel for loops look like serial for loops, their behavior is dramatically different. In particular, the iterations do not happen in a specified order, and writes to variables or arrays will not be globally visible since iterations run on different processes. Any variables used inside the parallel loop will be copied and broadcast to each process.
For example, the following code will not work as intended:
```julia
a = zeros(100000)
@distributed for i = 1:100000
    a[i] = i
end
```
This code will not initialize all of `a`, since each process will have a separate copy of it. Parallel for loops like these must be avoided. Fortunately, [Shared Arrays](#man-shared-arrays) can be used to get around this limitation:
```julia
using SharedArrays

a = SharedArray{Float64}(10)
@distributed for i = 1:10
    a[i] = i
end
```
Using "outside" variables in parallel loops is perfectly reasonable if the variables are read-only:
```julia
a = randn(1000)
@distributed (+) for i = 1:100000
    f(a[rand(1:end)])
end
```
Here each iteration applies `f` to a randomly-chosen sample from a vector `a` shared by all processes.


## [Parallel Map and Loops](#Parallel-Map-and-Loops)
As you could see, the reduction operator can be omitted if it is not needed. In that case, the loop executes asynchronously, i.e. it spawns independent tasks on all available workers and returns an array of [`Future`](../../stdlib/Distributed/#Distributed.Future) immediately without waiting for completion. The caller can wait for the [`Future`](../../stdlib/Distributed/#Distributed.Future) completions at a later point by calling [`fetch`](../../base/parallel/#Base.fetch-Tuple%7BTask%7D) on them, or wait for completion at the end of the loop by prefixing it with [`@sync`](../../base/parallel/#Base.@sync), like `@sync @distributed for`.
In some cases no reduction operator is needed, and we merely wish to apply a function to all integers in some range (or, more generally, to all elements in some collection). This is another useful operation called *parallel map*, implemented in Julia as the [`pmap`](../../stdlib/Distributed/#Distributed.pmap) function. For example, we could compute the singular values of several large random matrices in parallel as follows:
```julia-repl
julia> M = Matrix{Float64}[rand(1000,1000) for i = 1:10];

julia> pmap(svdvals, M);
```


## [Parallel Map and Loops](#Parallel-Map-and-Loops)
Julia's [`pmap`](../../stdlib/Distributed/#Distributed.pmap) is designed for the case where each function call does a large amount of work. In contrast, `@distributed for` can handle situations where each iteration is tiny, perhaps merely summing two numbers. Only worker processes are used by both [`pmap`](../../stdlib/Distributed/#Distributed.pmap) and `@distributed for` for the parallel computation. In case of `@distributed for`, the final reduction is done on the calling process.


## [Remote References and AbstractChannels](#Remote-References-and-AbstractChannels)
Remote references always refer to an implementation of an `AbstractChannel`.
A concrete implementation of an `AbstractChannel` (like `Channel`), is required to implement [`put!`](../../base/parallel/#Base.put!-Tuple%7BChannel,%20Any%7D), [`take!`](../../base/io-network/#Base.take!-Tuple%7BBase.GenericIOBuffer%7D), [`fetch`](../../base/parallel/#Base.fetch-Tuple%7BTask%7D), [`isready`](../../base/parallel/#Base.isready-Tuple%7BChannel%7D) and [`wait`](../../base/parallel/#Base.wait). The remote object referred to by a [`Future`](../../stdlib/Distributed/#Distributed.Future) is stored in a `Channel{Any}(1)`, i.e., a `Channel` of size 1 capable of holding objects of `Any` type.
[`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel), which is rewritable, can point to any type and size of channels, or any other implementation of an `AbstractChannel`.
The constructor `RemoteChannel(f::Function, pid)()` allows us to construct references to channels holding more than one value of a specific type. `f` is a function executed on `pid` and it must return an `AbstractChannel`.
For example, `RemoteChannel(()->Channel{Int}(10), pid)`, will return a reference to a channel of type `Int` and size 10. The channel exists on worker `pid`.


## [Remote References and AbstractChannels](#Remote-References-and-AbstractChannels)
Methods [`put!`](../../base/parallel/#Base.put!-Tuple%7BChannel,%20Any%7D), [`take!`](../../base/io-network/#Base.take!-Tuple%7BBase.GenericIOBuffer%7D), [`fetch`](../../base/parallel/#Base.fetch-Tuple%7BTask%7D), [`isready`](../../base/parallel/#Base.isready-Tuple%7BChannel%7D) and [`wait`](../../base/parallel/#Base.wait) on a [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel) are proxied onto the backing store on the remote process.
[`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel) can thus be used to refer to user implemented `AbstractChannel` objects. A simple example of this is the following `DictChannel` which uses a dictionary as its remote store:


## [Remote References and AbstractChannels](#Remote-References-and-AbstractChannels)
```julia-repl
julia> struct DictChannel{T} <: AbstractChannel{T}
           d::Dict
           cond_take::Threads.Condition    # waiting for data to become available
           DictChannel{T}() where {T} = new(Dict(), Threads.Condition())
           DictChannel() = DictChannel{Any}()
       end

julia> begin
       function Base.put!(D::DictChannel, k, v)
           @lock D.cond_take begin
               D.d[k] = v
               notify(D.cond_take)
           end
           return D
       end
       function Base.take!(D::DictChannel, k)
           @lock D.cond_take begin
               v = fetch(D, k)
               delete!(D.d, k)
               return v
           end
       end
       Base.isready(D::DictChannel) = @lock D.cond_take !isempty(D.d)
       Base.isready(D::DictChannel, k) = @lock D.cond_take haskey(D.d, k)
       function Base.fetch(D::DictChannel, k)
           @lock D.cond_take begin
               wait(D, k)
               return D.d[k]
           end
       end
       function Base.wait(D::DictChannel, k)
           @lock D.cond_take begin
               while !isready(D, k)
                   wait(D.cond_take)
               end
           end
       end
       end;

julia> d = DictChannel();

julia> isready(d)
false

julia> put!(d, :k, :v);

julia> isready(d, :k)
true

julia> fetch(d, :k)
:v

julia> wait(d, :k)

julia> take!(d, :k)
:v

julia> isready(d, :k)
false
```


## [Channels and RemoteChannels](#Channels-and-RemoteChannels)
-   A [`Channel`](../../base/parallel/#Base.Channel) is local to a process. Worker 2 cannot directly refer to a [`Channel`](../../base/parallel/#Base.Channel) on worker 3 and vice-versa. A [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel), however, can put and take values across workers.
-   A [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel) can be thought of as a *handle* to a [`Channel`](../../base/parallel/#Base.Channel).
-   The process id, `pid`, associated with a [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel) identifies the process where the backing store, i.e., the backing [`Channel`](../../base/parallel/#Base.Channel) exists.
-   Any process with a reference to a [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel) can put and take items from the channel. Data is automatically sent to (or retrieved from) the process a [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel) is associated with.
-   Serializing a [`Channel`](../../base/parallel/#Base.Channel) also serializes any data present in the channel. Deserializing it therefore effectively makes a copy of the original object.


## [Channels and RemoteChannels](#Channels-and-RemoteChannels)
-   On the other hand, serializing a [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel) only involves the serialization of an identifier that identifies the location and instance of [`Channel`](../../base/parallel/#Base.Channel) referred to by the handle. A deserialized [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel) object (on any worker), therefore also points to the same backing store as the original.
The channels example from above can be modified for interprocess communication, as shown below.
We start 4 workers to process a single `jobs` remote channel. Jobs, identified by an id (`job_id`), are written to the channel. Each remotely executing task in this simulation reads a `job_id`, waits for a random amount of time and writes back a tuple of `job_id`, time taken and its own `pid` to the results channel. Finally all the `results` are printed out on the master process.


## [Channels and RemoteChannels](#Channels-and-RemoteChannels)
```julia-repl
julia> addprocs(4); # add worker processes

julia> const jobs = RemoteChannel(()->Channel{Int}(32));

julia> const results = RemoteChannel(()->Channel{Tuple}(32));

julia> @everywhere function do_work(jobs, results) # define work function everywhere
           while true
               job_id = take!(jobs)
               exec_time = rand()
               sleep(exec_time) # simulates elapsed time doing actual work
               put!(results, (job_id, exec_time, myid()))
           end
       end

julia> function make_jobs(n)
           for i in 1:n
               put!(jobs, i)
           end
       end;

julia> n = 12;

julia> errormonitor(@async make_jobs(n)); # feed the jobs channel with "n" jobs

julia> for p in workers() # start tasks on the workers to process requests in parallel
           remote_do(do_work, p, jobs, results)
       end

julia> @elapsed while n > 0 # print out results
           job_id, exec_time, where = take!(results)
           println("$job_id finished in $(round(exec_time; digits=2)) seconds on worker $where")
           global n = n - 1
       end
1 finished in 0.18 seconds on worker 4
2 finished in 0.26 seconds on worker 5
6 finished in 0.12 seconds on worker 4
7 finished in 0.18 seconds on worker 4
5 finished in 0.35 seconds on worker 5
4 finished in 0.68 seconds on worker 2
3 finished in 0.73 seconds on worker 3
11 finished in 0.01 seconds on worker 3
12 finished in 0.02 seconds on worker 3
9 finished in 0.26 seconds on worker 5
8 finished in 0.57 seconds on worker 4
10 finished in 0.58 seconds on worker 2
0.055971741
```


### [Remote References and Distributed Garbage Collection](#Remote-References-and-Distributed-Garbage-Collection)
Objects referred to by remote references can be freed only when *all* held references in the cluster are deleted.
The node where the value is stored keeps track of which of the workers have a reference to it. Every time a [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel) or a (unfetched) [`Future`](../../stdlib/Distributed/#Distributed.Future) is serialized to a worker, the node pointed to by the reference is notified. And every time a [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel) or a (unfetched) [`Future`](../../stdlib/Distributed/#Distributed.Future) is garbage collected locally, the node owning the value is again notified. This is implemented in an internal cluster aware serializer. Remote references are only valid in the context of a running cluster. Serializing and deserializing references to and from regular `IO` objects is not supported.
The notifications are done via sending of "tracking" messages–an "add reference" message when a reference is serialized to a different process and a "delete reference" message when a reference is locally garbage collected.
Since [`Future`](../../stdlib/Distributed/#Distributed.Future)s are write-once and cached locally, the act of [`fetch`](../../base/parallel/#Base.fetch-Tuple%7BTask%7D)ing a [`Future`](../../stdlib/Distributed/#Distributed.Future) also updates reference tracking information on the node owning the value.


### [Remote References and Distributed Garbage Collection](#Remote-References-and-Distributed-Garbage-Collection)
The node which owns the value frees it once all references to it are cleared.
With [`Future`](../../stdlib/Distributed/#Distributed.Future)s, serializing an already fetched [`Future`](../../stdlib/Distributed/#Distributed.Future) to a different node also sends the value since the original remote store may have collected the value by this time.
It is important to note that *when* an object is locally garbage collected depends on the size of the object and the current memory pressure in the system.
In case of remote references, the size of the local reference object is quite small, while the value stored on the remote node may be quite large. Since the local object may not be collected immediately, it is a good practice to explicitly call [`finalize`](../../base/base/#Base.finalize) on local instances of a [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel), or on unfetched [`Future`](../../stdlib/Distributed/#Distributed.Future)s. Since calling [`fetch`](../../base/parallel/#Base.fetch-Tuple%7BTask%7D) on a [`Future`](../../stdlib/Distributed/#Distributed.Future) also removes its reference from the remote store, this is not required on fetched [`Future`](../../stdlib/Distributed/#Distributed.Future)s. Explicitly calling [`finalize`](../../base/base/#Base.finalize) results in an immediate message sent to the remote node to go ahead and remove its reference to the value.


### [Remote References and Distributed Garbage Collection](#Remote-References-and-Distributed-Garbage-Collection)
Once finalized, a reference becomes invalid and cannot be used in any further calls.


## [Local invocations](#Local-invocations)
Data is necessarily copied over to the remote node for execution. This is the case for both remotecalls and when data is stored to a [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel) / [`Future`](../../stdlib/Distributed/#Distributed.Future) on a different node. As expected, this results in a copy of the serialized objects on the remote node. However, when the destination node is the local node, i.e. the calling process id is the same as the remote node id, it is executed as a local call. It is usually (not always) executed in a different task - but there is no serialization/deserialization of data. Consequently, the call refers to the same object instances as passed - no copies are created. This behavior is highlighted below:


## [Local invocations](#Local-invocations)
```julia-repl
julia> using Distributed;

julia> rc = RemoteChannel(()->Channel(3));   # RemoteChannel created on local node

julia> v = [0];

julia> for i in 1:3
           v[1] = i                          # Reusing `v`
           put!(rc, v)
       end;

julia> result = [take!(rc) for _ in 1:3];

julia> println(result);
Array{Int64,1}[[3], [3], [3]]

julia> println("Num Unique objects : ", length(unique(map(objectid, result))));
Num Unique objects : 1

julia> addprocs(1);

julia> rc = RemoteChannel(()->Channel(3), workers()[1]);   # RemoteChannel created on remote node

julia> v = [0];

julia> for i in 1:3
           v[1] = i
           put!(rc, v)
       end;

julia> result = [take!(rc) for _ in 1:3];

julia> println(result);
Array{Int64,1}[[1], [2], [3]]

julia> println("Num Unique objects : ", length(unique(map(objectid, result))));
Num Unique objects : 3
```
As can be seen, [`put!`](../../base/parallel/#Base.put!-Tuple%7BChannel,%20Any%7D) on a locally owned [`RemoteChannel`](../../stdlib/Distributed/#Distributed.RemoteChannel) with the same object `v` modified between calls results in the same single object instance stored. As opposed to copies of `v` being created when the node owning `rc` is a different node.
It is to be noted that this is generally not an issue. It is something to be factored in only if the object is both being stored locally and modified post the call. In such cases it may be appropriate to store a `deepcopy` of the object.


## [Local invocations](#Local-invocations)
This is also true for remotecalls on the local node as seen in the following example:
```julia-repl
julia> using Distributed; addprocs(1);

julia> v = [0];

julia> v2 = remotecall_fetch(x->(x[1] = 1; x), myid(), v);     # Executed on local node

julia> println("v=$v, v2=$v2, ", v === v2);
v=[1], v2=[1], true

julia> v = [0];

julia> v2 = remotecall_fetch(x->(x[1] = 1; x), workers()[1], v); # Executed on remote node

julia> println("v=$v, v2=$v2, ", v === v2);
v=[0], v2=[1], false
```
As can be seen once again, a remote call onto the local node behaves just like a direct invocation. The call modifies local objects passed as arguments. In the remote invocation, it operates on a copy of the arguments.
To repeat, in general this is not an issue. If the local node is also being used as a compute node, and the arguments used post the call, this behavior needs to be factored in and if required deep copies of arguments must be passed to the call invoked on the local node. Calls on remote nodes will always operate on copies of arguments.


## [Shared Arrays](#man-shared-arrays)
Shared Arrays use system shared memory to map the same array across many processes. A [`SharedArray`](../../stdlib/SharedArrays/#SharedArrays.SharedArray) is a good choice when you want to have a large amount of data jointly accessible to two or more processes on the same machine. Shared Array support is available via the module `SharedArrays`, which must be explicitly loaded on all participating workers.
A complementary data structure is provided by the external package [`DistributedArrays.jl`](https://github.com/JuliaParallel/DistributedArrays.jl) in the form of a `DArray`. While there are some similarities to a [`SharedArray`](../../stdlib/SharedArrays/#SharedArrays.SharedArray), the behavior of a [`DArray`](https://github.com/JuliaParallel/DistributedArrays.jl) is quite different. In a [`SharedArray`](../../stdlib/SharedArrays/#SharedArrays.SharedArray), each "participating" process has access to the entire array; in contrast, in a [`DArray`](https://github.com/JuliaParallel/DistributedArrays.jl), each process has local access to just a chunk of the data, and no two processes share the same chunk.


## [Shared Arrays](#man-shared-arrays)
[`SharedArray`](../../stdlib/SharedArrays/#SharedArrays.SharedArray) indexing (assignment and accessing values) works just as with regular arrays, and is efficient because the underlying memory is available to the local process. Therefore, most algorithms work naturally on [`SharedArray`](../../stdlib/SharedArrays/#SharedArrays.SharedArray)s, albeit in single-process mode. In cases where an algorithm insists on an [`Array`](../../base/arrays/#Core.Array) input, the underlying array can be retrieved from a [`SharedArray`](../../stdlib/SharedArrays/#SharedArrays.SharedArray) by calling [`sdata`](../../stdlib/SharedArrays/#SharedArrays.sdata). For other `AbstractArray` types, [`sdata`](../../stdlib/SharedArrays/#SharedArrays.sdata) just returns the object itself, so it's safe to use [`sdata`](../../stdlib/SharedArrays/#SharedArrays.sdata) on any `Array`-type object.
The constructor for a shared array is of the form:
```julia
SharedArray{T,N}(dims::NTuple; init=false, pids=Int[])
```
which creates an `N`-dimensional shared array of a bits type `T` and size `dims` across the processes specified by `pids`. Unlike distributed arrays, a shared array is accessible only from those participating workers specified by the `pids` named argument (and the creating process too, if it is on the same host). Note that only elements that are [`isbits`](../../base/base/#Base.isbits) are supported in a SharedArray.


## [Shared Arrays](#man-shared-arrays)
If an `init` function, of signature `initfn(S::SharedArray)`, is specified, it is called on all the participating workers. You can specify that each worker runs the `init` function on a distinct portion of the array, thereby parallelizing initialization.
Here's a brief example:
```julia-repl
julia> using Distributed

julia> addprocs(3)
3-element Array{Int64,1}:
 2
 3
 4

julia> @everywhere using SharedArrays

julia> S = SharedArray{Int,2}((3,4), init = S -> S[localindices(S)] = repeat([myid()], length(localindices(S))))
3×4 SharedArray{Int64,2}:
 2  2  3  4
 2  3  3  4
 2  3  4  4

julia> S[3,2] = 7
7

julia> S
3×4 SharedArray{Int64,2}:
 2  2  3  4
 2  3  3  4
 2  7  4  4
```
[`SharedArrays.localindices`](../../stdlib/SharedArrays/#SharedArrays.localindices) provides disjoint one-dimensional ranges of indices, and is sometimes convenient for splitting up tasks among processes. You can, of course, divide the work any way you wish:
```julia-repl
julia> S = SharedArray{Int,2}((3,4), init = S -> S[indexpids(S):length(procs(S)):length(S)] = repeat([myid()], length( indexpids(S):length(procs(S)):length(S))))
3×4 SharedArray{Int64,2}:
 2  2  2  2
 3  3  3  3
 4  4  4  4
```
Since all processes have access to the underlying data, you do have to be careful not to set up conflicts. For example:
```julia
@sync begin
    for p in procs(S)
        @async begin
            remotecall_wait(fill!, p, S, p)
        end
    end
end
```


## [Shared Arrays](#man-shared-arrays)
would result in undefined behavior. Because each process fills the *entire* array with its own `pid`, whichever process is the last to execute (for any particular element of `S`) will have its `pid` retained.
As a more extended and complex example, consider running the following "kernel" in parallel:
```julia
q[i,j,t+1] = q[i,j,t] + u[i,j,t]
```
In this case, if we try to split up the work using a one-dimensional index, we are likely to run into trouble: if `q[i,j,t]` is near the end of the block assigned to one worker and `q[i,j,t+1]` is near the beginning of the block assigned to another, it's very likely that `q[i,j,t]` will not be ready at the time it's needed for computing `q[i,j,t+1]`. In such cases, one is better off chunking the array manually. Let's split along the second dimension. Define a function that returns the `(irange, jrange)` indices assigned to this worker:
```julia-repl
julia> @everywhere function myrange(q::SharedArray)
           idx = indexpids(q)
           if idx == 0 # This worker is not assigned a piece
               return 1:0, 1:0
           end
           nchunks = length(procs(q))
           splits = [round(Int, s) for s in range(0, stop=size(q,2), length=nchunks+1)]
           1:size(q,1), splits[idx]+1:splits[idx+1]
       end
```
Next, define the kernel:


## [Shared Arrays](#man-shared-arrays)
```julia-repl
julia> @everywhere function advection_chunk!(q, u, irange, jrange, trange)
           @show (irange, jrange, trange)  # display so we can see what's happening
           for t in trange, j in jrange, i in irange
               q[i,j,t+1] = q[i,j,t] + u[i,j,t]
           end
           q
       end
```
We also define a convenience wrapper for a `SharedArray` implementation
```julia-repl
julia> @everywhere advection_shared_chunk!(q, u) =
           advection_chunk!(q, u, myrange(q)..., 1:size(q,3)-1)
```
Now let's compare three different versions, one that runs in a single process:
```julia-repl
julia> advection_serial!(q, u) = advection_chunk!(q, u, 1:size(q,1), 1:size(q,2), 1:size(q,3)-1);
```
one that uses [`@distributed`](../../stdlib/Distributed/#Distributed.@distributed):
```julia-repl
julia> function advection_parallel!(q, u)
           for t = 1:size(q,3)-1
               @sync @distributed for j = 1:size(q,2)
                   for i = 1:size(q,1)
                       q[i,j,t+1]= q[i,j,t] + u[i,j,t]
                   end
               end
           end
           q
       end;
```
and one that delegates in chunks:
```julia-repl
julia> function advection_shared!(q, u)
           @sync begin
               for p in procs(q)
                   @async remotecall_wait(advection_shared_chunk!, p, q, u)
               end
           end
           q
       end;
```
If we create `SharedArray`s and time these functions, we get the following results (with `julia -p 4`):


## [Shared Arrays](#man-shared-arrays)
```julia-repl
julia> q = SharedArray{Float64,3}((500,500,500));

julia> u = SharedArray{Float64,3}((500,500,500));
```
Run the functions once to JIT-compile and [`@time`](../profile/#@time) them on the second run:
```julia-repl
julia> @time advection_serial!(q, u);
(irange,jrange,trange) = (1:500,1:500,1:499)
 830.220 milliseconds (216 allocations: 13820 bytes)

julia> @time advection_parallel!(q, u);
   2.495 seconds      (3999 k allocations: 289 MB, 2.09% gc time)

julia> @time advection_shared!(q,u);
        From worker 2:       (irange,jrange,trange) = (1:500,1:125,1:499)
        From worker 4:       (irange,jrange,trange) = (1:500,251:375,1:499)
        From worker 3:       (irange,jrange,trange) = (1:500,126:250,1:499)
        From worker 5:       (irange,jrange,trange) = (1:500,376:500,1:499)
 238.119 milliseconds (2264 allocations: 169 KB)
```
The biggest advantage of `advection_shared!` is that it minimizes traffic among the workers, allowing each to compute for an extended time on the assigned piece.


### [Shared Arrays and Distributed Garbage Collection](#Shared-Arrays-and-Distributed-Garbage-Collection)
Like remote references, shared arrays are also dependent on garbage collection on the creating node to release references from all participating workers. Code which creates many short lived shared array objects would benefit from explicitly finalizing these objects as soon as possible. This results in both memory and file handles mapping the shared segment being released sooner.


## [ClusterManagers](#ClusterManagers)
The launching, management and networking of Julia processes into a logical cluster is done via cluster managers. A `ClusterManager` is responsible for
-   launching worker processes in a cluster environment
-   managing events during the lifetime of each worker
-   optionally, providing data transport
A Julia cluster has the following characteristics:
-   The initial Julia process, also called the `master`, is special and has an `id` of 1.
-   Only the `master` process can add or remove worker processes.
-   All processes can directly communicate with each other.
Connections between workers (using the in-built TCP/IP transport) is established in the following manner:
-   [`addprocs`](../../stdlib/Distributed/#Distributed.addprocs) is called on the master process with a `ClusterManager` object.
-   [`addprocs`](../../stdlib/Distributed/#Distributed.addprocs) calls the appropriate [`launch`](../../stdlib/Distributed/#Distributed.launch) method which spawns required number of worker processes on appropriate machines.
-   Each worker starts listening on a free port and writes out its host and port information to [`stdout`](../../base/io-network/#Base.stdout).
-   The cluster manager captures the [`stdout`](../../base/io-network/#Base.stdout) of each worker and makes it available to the master process.
-   The master process parses this information and sets up TCP/IP connections to each worker.
-   Every worker is also notified of other workers in the cluster.


## [ClusterManagers](#ClusterManagers)
-   Each worker connects to all workers whose `id` is less than the worker's own `id`.
-   In this way a mesh network is established, wherein every worker is directly connected with every other worker.
While the default transport layer uses plain [`TCPSocket`](../../stdlib/Sockets/#Sockets.TCPSocket), it is possible for a Julia cluster to provide its own transport.
Julia provides two in-built cluster managers:
-   `LocalManager`, used when [`addprocs()`](../../stdlib/Distributed/#Distributed.addprocs) or [`addprocs(np::Integer)`](../../stdlib/Distributed/#Distributed.addprocs) are called
-   `SSHManager`, used when [`addprocs(hostnames::Array)`](../../stdlib/Distributed/#Distributed.addprocs) is called with a list of hostnames
`LocalManager` is used to launch additional workers on the same host, thereby leveraging multi-core and multi-processor hardware.
Thus, a minimal cluster manager would need to:
-   be a subtype of the abstract `ClusterManager`
-   implement [`launch`](../../stdlib/Distributed/#Distributed.launch), a method responsible for launching new workers
-   implement [`manage`](../../stdlib/Distributed/#Distributed.manage), which is called at various events during a worker's lifetime (for example, sending an interrupt signal)
[`addprocs(manager::FooManager)`](../../stdlib/Distributed/#Distributed.addprocs) requires `FooManager` to implement:


## [ClusterManagers](#ClusterManagers)
```julia
function launch(manager::FooManager, params::Dict, launched::Array, c::Condition)
    [...]
end

function manage(manager::FooManager, id::Integer, config::WorkerConfig, op::Symbol)
    [...]
end
```
As an example let us see how the `LocalManager`, the manager responsible for starting workers on the same host, is implemented:
```julia
struct LocalManager <: ClusterManager
    np::Integer
end

function launch(manager::LocalManager, params::Dict, launched::Array, c::Condition)
    [...]
end

function manage(manager::LocalManager, id::Integer, config::WorkerConfig, op::Symbol)
    [...]
end
```
The [`launch`](../../stdlib/Distributed/#Distributed.launch) method takes the following arguments:
-   `manager::ClusterManager`: the cluster manager that [`addprocs`](../../stdlib/Distributed/#Distributed.addprocs) is called with
-   `params::Dict`: all the keyword arguments passed to [`addprocs`](../../stdlib/Distributed/#Distributed.addprocs)
-   `launched::Array`: the array to append one or more `WorkerConfig` objects to
-   `c::Condition`: the condition variable to be notified as and when workers are launched
The [`launch`](../../stdlib/Distributed/#Distributed.launch) method is called asynchronously in a separate task. The termination of this task signals that all requested workers have been launched. Hence the [`launch`](../../stdlib/Distributed/#Distributed.launch) function MUST exit as soon as all the requested workers have been launched.


## [ClusterManagers](#ClusterManagers)
Newly launched workers are connected to each other and the master process in an all-to-all manner. Specifying the command line argument `--worker[=<cookie>]` results in the launched processes initializing themselves as workers and connections being set up via TCP/IP sockets.
All workers in a cluster share the same [cookie](#man-cluster-cookie) as the master. When the cookie is unspecified, i.e, with the `--worker` option, the worker tries to read it from its standard input. `LocalManager` and `SSHManager` both pass the cookie to newly launched workers via their standard inputs.
By default a worker will listen on a free port at the address returned by a call to [`getipaddr()`](../../stdlib/Sockets/#Sockets.getipaddr). A specific address to listen on may be specified by optional argument `--bind-to bind_addr[:port]`. This is useful for multi-homed hosts.
As an example of a non-TCP/IP transport, an implementation may choose to use MPI, in which case `--worker` must NOT be specified. Instead, newly launched workers should call `init_worker(cookie)` before using any of the parallel constructs.
For every worker launched, the [`launch`](../../stdlib/Distributed/#Distributed.launch) method must add a `WorkerConfig` object (with appropriate fields initialized) to `launched`


## [ClusterManagers](#ClusterManagers)
```julia
mutable struct WorkerConfig
    # Common fields relevant to all cluster managers
    io::Union{IO, Nothing}
    host::Union{AbstractString, Nothing}
    port::Union{Integer, Nothing}

    # Used when launching additional workers at a host
    count::Union{Int, Symbol, Nothing}
    exename::Union{AbstractString, Cmd, Nothing}
    exeflags::Union{Cmd, Nothing}

    # External cluster managers can use this to store information at a per-worker level
    # Can be a dict if multiple fields need to be stored.
    userdata::Any

    # SSHManager / SSH tunnel connections to workers
    tunnel::Union{Bool, Nothing}
    bind_addr::Union{AbstractString, Nothing}
    sshflags::Union{Cmd, Nothing}
    max_parallel::Union{Integer, Nothing}

    # Used by Local/SSH managers
    connect_at::Any

    [...]
end
```
Most of the fields in `WorkerConfig` are used by the inbuilt managers. Custom cluster managers would typically specify only `io` or `host` / `port`:
-   If `io` is specified, it is used to read host/port information. A Julia worker prints out its bind address and port at startup. This allows Julia workers to listen on any free port available instead of requiring worker ports to be configured manually.
-   If `io` is not specified, `host` and `port` are used to connect.
-   `count`, `exename` and `exeflags` are relevant for launching additional workers from a worker. For example, a cluster manager may launch a single worker per node, and use that to launch additional workers.


## [ClusterManagers](#ClusterManagers)
    -   `count` with an integer value `n` will launch a total of `n` workers.
    -   `count` with a value of `:auto` will launch as many workers as the number of CPU threads (logical cores) on that machine.
    -   `exename` is the name of the `julia` executable including the full path.
    -   `exeflags` should be set to the required command line arguments for new workers.
-   `tunnel`, `bind_addr`, `sshflags` and `max_parallel` are used when a ssh tunnel is required to connect to the workers from the master process.
-   `userdata` is provided for custom cluster managers to store their own worker-specific information.
`manage(manager::FooManager, id::Integer, config::WorkerConfig, op::Symbol)` is called at different times during the worker's lifetime with appropriate `op` values:
-   with `:register`/`:deregister` when a worker is added / removed from the Julia worker pool.
-   with `:interrupt` when `interrupt(workers)` is called. The `ClusterManager` should signal the appropriate worker with an interrupt signal.
-   with `:finalize` for cleanup purposes.


### [Cluster Managers with Custom Transports](#Cluster-Managers-with-Custom-Transports)
Replacing the default TCP/IP all-to-all socket connections with a custom transport layer is a little more involved. Each Julia process has as many communication tasks as the workers it is connected to. For example, consider a Julia cluster of 32 processes in an all-to-all mesh network:
-   Each Julia process thus has 31 communication tasks.
-   Each task handles all incoming messages from a single remote worker in a message-processing loop.
-   The message-processing loop waits on an `IO` object (for example, a [`TCPSocket`](../../stdlib/Sockets/#Sockets.TCPSocket) in the default implementation), reads an entire message, processes it and waits for the next one.
-   Sending messages to a process is done directly from any Julia task–not just communication tasks–again, via the appropriate `IO` object.
Replacing the default transport requires the new implementation to set up connections to remote workers and to provide appropriate `IO` objects that the message-processing loops can wait on. The manager-specific callbacks to be implemented are:
```julia
connect(manager::FooManager, pid::Integer, config::WorkerConfig)
kill(manager::FooManager, pid::Int, config::WorkerConfig)
```
The default implementation (which uses TCP/IP sockets) is implemented as `connect(manager::ClusterManager, pid::Integer, config::WorkerConfig)`.


### [Cluster Managers with Custom Transports](#Cluster-Managers-with-Custom-Transports)
`connect` should return a pair of `IO` objects, one for reading data sent from worker `pid`, and the other to write data that needs to be sent to worker `pid`. Custom cluster managers can use an in-memory `BufferStream` as the plumbing to proxy data between the custom, possibly non-`IO` transport and Julia's in-built parallel infrastructure.
A `BufferStream` is an in-memory [`IOBuffer`](../../base/io-network/#Base.IOBuffer) which behaves like an `IO`–it is a stream which can be handled asynchronously.
The folder `clustermanager/0mq` in the [Examples repository](https://github.com/JuliaAttic/Examples) contains an example of using ZeroMQ to connect Julia workers in a star topology with a 0MQ broker in the middle. Note: The Julia processes are still all *logically* connected to each other–any worker can message any other worker directly without any awareness of 0MQ being used as the transport layer.
When using custom transports:
-   Julia workers must NOT be started with `--worker`. Starting with `--worker` will result in the newly launched workers defaulting to the TCP/IP socket transport implementation.
-   For every incoming logical connection with a worker, `Base.process_messages(rd::IO, wr::IO)()` must be called. This launches a new task that handles reading and writing of messages from/to the worker represented by the `IO` objects.
-   `init_worker(cookie, manager::FooManager)` *must* be called as part of worker process initialization.


### [Cluster Managers with Custom Transports](#Cluster-Managers-with-Custom-Transports)
-   Field `connect_at::Any` in `WorkerConfig` can be set by the cluster manager when [`launch`](../../stdlib/Distributed/#Distributed.launch) is called. The value of this field is passed in all [`connect`](../../stdlib/Distributed/#Sockets.connect-Tuple%7BClusterManager,%20Int64,%20WorkerConfig%7D) callbacks. Typically, it carries information on *how to connect* to a worker. For example, the TCP/IP socket transport uses this field to specify the `(host, port)` tuple at which to connect to a worker.
`kill(manager, pid, config)` is called to remove a worker from the cluster. On the master process, the corresponding `IO` objects must be closed by the implementation to ensure proper cleanup. The default implementation simply executes an `exit()` call on the specified remote worker.
The Examples folder `clustermanager/simple` is an example that shows a simple implementation using UNIX domain sockets for cluster setup.


### [Network Requirements for LocalManager and SSHManager](#Network-Requirements-for-LocalManager-and-SSHManager)
Julia clusters are designed to be executed on already secured environments on infrastructure such as local laptops, departmental clusters, or even the cloud. This section covers network security requirements for the inbuilt `LocalManager` and `SSHManager`:
-   The master process does not listen on any port. It only connects out to the workers.
-   Each worker binds to only one of the local interfaces and listens on an ephemeral port number assigned by the OS.
-   `LocalManager`, used by `addprocs(N)`, by default binds only to the loopback interface. This means that workers started later on remote hosts (or by anyone with malicious intentions) are unable to connect to the cluster. An `addprocs(4)` followed by an `addprocs(["remote_host"])` will fail. Some users may need to create a cluster comprising their local system and a few remote systems. This can be done by explicitly requesting `LocalManager` to bind to an external network interface via the `restrict` keyword argument: `addprocs(4; restrict=false)`.
-   `SSHManager`, used by `addprocs(list_of_remote_hosts)`, launches workers on remote hosts via SSH. By default SSH is only used to launch Julia workers. Subsequent master-worker and worker-worker connections use plain, unencrypted TCP/IP sockets. The remote hosts must have passwordless login enabled. Additional SSH flags or credentials may be specified via keyword argument `sshflags`.


### [Network Requirements for LocalManager and SSHManager](#Network-Requirements-for-LocalManager-and-SSHManager)
-   `addprocs(list_of_remote_hosts; tunnel=true, sshflags=<ssh keys and other flags>)` is useful when we wish to use SSH connections for master-worker too. A typical scenario for this is a local laptop running the Julia REPL (i.e., the master) with the rest of the cluster on the cloud, say on Amazon EC2. In this case only port 22 needs to be opened at the remote cluster coupled with SSH client authenticated via public key infrastructure (PKI). Authentication credentials can be supplied via `sshflags`, for example `` sshflags=`-i <keyfile>` ``.
    In an all-to-all topology (the default), all workers connect to each other via plain TCP sockets. The security policy on the cluster nodes must thus ensure free connectivity between workers for the ephemeral port range (varies by OS).
    Securing and encrypting all worker-worker traffic (via SSH) or encrypting individual messages can be done via a custom `ClusterManager`.


### [Network Requirements for LocalManager and SSHManager](#Network-Requirements-for-LocalManager-and-SSHManager)
-   If you specify `multiplex=true` as an option to [`addprocs`](../../stdlib/Distributed/#Distributed.addprocs), SSH multiplexing is used to create a tunnel between the master and workers. If you have configured SSH multiplexing on your own and the connection has already been established, SSH multiplexing is used regardless of `multiplex` option. If multiplexing is enabled, forwarding is set by using the existing connection (`-O forward` option in ssh). This is beneficial if your servers require password authentication; you can avoid authentication in Julia by logging in to the server ahead of [`addprocs`](../../stdlib/Distributed/#Distributed.addprocs). The control socket will be located at `~/.ssh/julia-%r@%h:%p` during the session unless the existing multiplexing connection is used. Note that bandwidth may be limited if you create multiple processes on a node and enable multiplexing, because in that case processes share a single multiplexing TCP connection.


### [Cluster Cookie](#man-cluster-cookie)
All processes in a cluster share the same cookie which, by default, is a randomly generated string on the master process:
-   [`cluster_cookie()`](../../stdlib/Distributed/#Distributed.cluster_cookie-Tuple%7B%7D) returns the cookie, while `cluster_cookie(cookie)()` sets it and returns the new cookie.
-   All connections are authenticated on both sides to ensure that only workers started by the master are allowed to connect to each other.
-   The cookie may be passed to the workers at startup via argument `--worker=<cookie>`. If argument `--worker` is specified without the cookie, the worker tries to read the cookie from its standard input ([`stdin`](../../base/io-network/#Base.stdin)). The `stdin` is closed immediately after the cookie is retrieved.
-   `ClusterManager`s can retrieve the cookie on the master by calling [`cluster_cookie()`](../../stdlib/Distributed/#Distributed.cluster_cookie-Tuple%7B%7D). Cluster managers not using the default TCP/IP transport (and hence not specifying `--worker`) must call `init_worker(cookie, manager)` with the same cookie as on the master.
Note that environments requiring higher levels of security can implement this via a custom `ClusterManager`. For example, cookies can be pre-shared and hence not specified as a startup argument.


## [Specifying Network Topology (Experimental)](#Specifying-Network-Topology-(Experimental))
The keyword argument `topology` passed to [`addprocs`](../../stdlib/Distributed/#Distributed.addprocs) is used to specify how the workers must be connected to each other:
-   `:all_to_all`, the default: all workers are connected to each other.
-   `:master_worker`: only the driver process, i.e. `pid` 1, has connections to the workers.
-   `:custom`: the `launch` method of the cluster manager specifies the connection topology via the fields `ident` and `connect_idents` in `WorkerConfig`. A worker with a cluster-manager-provided identity `ident` will connect to all workers specified in `connect_idents`.
Keyword argument `lazy=true|false` only affects `topology` option `:all_to_all`. If `true`, the cluster starts off with the master connected to all workers. Specific worker-worker connections are established at the first remote invocation between two workers. This helps in reducing initial resources allocated for intra-cluster communication. Connections are setup depending on the runtime requirements of a parallel program. Default value for `lazy` is `true`.
Currently, sending a message between unconnected workers results in an error. This behaviour, as with the functionality and interface, should be considered experimental in nature and may change in future releases.


## [Noteworthy external packages](#Noteworthy-external-packages)
Outside of Julia parallelism there are plenty of external packages that should be mentioned. For example, [`MPI.jl`](https://github.com/JuliaParallel/MPI.jl) is a Julia wrapper for the `MPI` protocol, [`Dagger.jl`](https://github.com/JuliaParallel/Dagger.jl) provides functionality similar to Python's [Dask](https://dask.org/), and [`DistributedArrays.jl`](https://github.com/JuliaParallel/Distributedarrays.jl) provides array operations distributed across workers, as [outlined above](#man-shared-arrays).
A mention must be made of Julia's GPU programming ecosystem, which includes:
1.  [CUDA.jl](https://github.com/JuliaGPU/CUDA.jl) wraps the various CUDA libraries and supports compiling Julia kernels for Nvidia GPUs.
2.  [oneAPI.jl](https://github.com/JuliaGPU/oneAPI.jl) wraps the oneAPI unified programming model, and supports executing Julia kernels on supported accelerators. Currently only Linux is supported.
3.  [AMDGPU.jl](https://github.com/JuliaGPU/AMDGPU.jl) wraps the AMD ROCm libraries and supports compiling Julia kernels for AMD GPUs. Currently only Linux is supported.
4.  High-level libraries like [KernelAbstractions.jl](https://github.com/JuliaGPU/KernelAbstractions.jl), [Tullio.jl](https://github.com/mcabbott/Tullio.jl) and [ArrayFire.jl](https://github.com/JuliaComputing/ArrayFire.jl).
In the following example we will use both `DistributedArrays.jl` and `CUDA.jl` to distribute an array across multiple processes by first casting it through `distribute()` and `CuArray()`.


## [Noteworthy external packages](#Noteworthy-external-packages)
Remember when importing `DistributedArrays.jl` to import it across all processes using [`@everywhere`](../../stdlib/Distributed/#Distributed.@everywhere)
```julia-repl
$ ./julia -p 4

julia> addprocs()

julia> @everywhere using DistributedArrays

julia> using CUDA

julia> B = ones(10_000) ./ 2;

julia> A = ones(10_000) .* π;

julia> C = 2 .* A ./ B;

julia> all(C .≈ 4*π)
true

julia> typeof(C)
Array{Float64,1}

julia> dB = distribute(B);

julia> dA = distribute(A);

julia> dC = 2 .* dA ./ dB;

julia> all(dC .≈ 4*π)
true

julia> typeof(dC)
DistributedArrays.DArray{Float64,1,Array{Float64,1}}

julia> cuB = CuArray(B);

julia> cuA = CuArray(A);

julia> cuC = 2 .* cuA ./ cuB;

julia> all(cuC .≈ 4*π);
true

julia> typeof(cuC)
CuArray{Float64,1}
```
In the following example we will use both `DistributedArrays.jl` and `CUDA.jl` to distribute an array across multiple processes and call a generic function on it.
```julia
function power_method(M, v)
    for i in 1:100
        v = M*v
        v /= norm(v)
    end

    return v, norm(M*v) / norm(v)  # or  (M*v) ./ v
end
```
`power_method` repeatedly creates a new vector and normalizes it. We have not specified any type signature in function declaration, let's see if it works with the aforementioned datatypes:


## [Noteworthy external packages](#Noteworthy-external-packages)
```julia-repl
julia> M = [2. 1; 1 1];

julia> v = rand(2)
2-element Array{Float64,1}:
0.40395
0.445877

julia> power_method(M,v)
([0.850651, 0.525731], 2.618033988749895)

julia> cuM = CuArray(M);

julia> cuv = CuArray(v);

julia> curesult = power_method(cuM, cuv);

julia> typeof(curesult)
CuArray{Float64,1}

julia> dM = distribute(M);

julia> dv = distribute(v);

julia> dC = power_method(dM, dv);

julia> typeof(dC)
Tuple{DistributedArrays.DArray{Float64,1,Array{Float64,1}},Float64}
```
To end this short exposure to external packages, we can consider `MPI.jl`, a Julia wrapper of the MPI protocol. As it would take too long to consider every inner function, it would be better to simply appreciate the approach used to implement the protocol.
Consider this toy script which simply calls each subprocess, instantiate its rank and when the master process is reached, performs the ranks' sum
```julia
import MPI

MPI.Init()

comm = MPI.COMM_WORLD
MPI.Barrier(comm)

root = 0
r = MPI.Comm_rank(comm)

sr = MPI.Reduce(r, MPI.SUM, root, comm)

if(MPI.Comm_rank(comm) == root)
   @printf("sum of ranks: %s\n", sr)
end

MPI.Finalize()
```
```julia
mpirun -np 4 ./julia example.jl
```


## [Noteworthy external packages](#Noteworthy-external-packages)
-   [1](#citeref-1)In this context, MPI refers to the MPI-1 standard. Beginning with MPI-2, the MPI standards committee introduced a new set of communication mechanisms, collectively referred to as Remote Memory Access (RMA). The motivation for adding rma to the MPI standard was to facilitate one-sided communication patterns. For additional information on the latest MPI standard, see <https://mpi-forum.org/docs>.
------------------------------------------------------------------------


# Running External Programs · The Julia Language
Source: https://docs.julialang.org/en/v1/manual/running-external-programs/


# [Running External Programs](#Running-External-Programs)
Julia borrows backtick notation for commands from the shell, Perl, and Ruby. However, in Julia, writing
```julia-repl
julia> `echo hello`
`echo hello`
```
differs in several aspects from the behavior in various shells, Perl, or Ruby:
-   Instead of immediately running the command, backticks create a [`Cmd`](../../base/base/#Base.Cmd) object to represent the command. You can use this object to connect the command to others via pipes, [`run`](../../base/base/#Base.run) it, and [`read`](../../base/io-network/#Base.read) or [`write`](../../base/io-network/#Base.write) to it.
-   When the command is run, Julia does not capture its output unless you specifically arrange for it to. Instead, the output of the command by default goes to [`stdout`](../../base/io-network/#Base.stdout) as it would using `libc`'s `system` call.
-   The command is never run with a shell. Instead, Julia parses the command syntax directly, appropriately interpolating variables and splitting on words as the shell would, respecting shell quoting syntax. The command is run as `julia`'s immediate child process, using `fork` and `exec` calls.
The following assumes a Posix environment as on Linux or MacOS. On Windows, many similar commands, such as `echo` and `dir`, are not external programs and instead are built into the shell `cmd.exe` itself. One option to run these commands is to invoke `cmd.exe`, for example `cmd /C echo hello`. Alternatively Julia can be run inside a Posix environment such as Cygwin.


# [Running External Programs](#Running-External-Programs)
Here's a simple example of running an external program:
```julia-repl
julia> mycommand = `echo hello`
`echo hello`

julia> typeof(mycommand)
Cmd

julia> run(mycommand);
hello
```
The `hello` is the output of the `echo` command, sent to [`stdout`](../../base/io-network/#Base.stdout). If the external command fails to run successfully, the run method throws an [`ProcessFailedException`](../../base/base/#Base.ProcessFailedException).
If you want to read the output of the external command, [`read`](../../base/io-network/#Base.read) or [`readchomp`](../../base/io-network/#Base.readchomp) can be used instead:
```julia-repl
julia> read(`echo hello`, String)
"hello\n"

julia> readchomp(`echo hello`)
"hello"
```
More generally, you can use [`open`](../../base/io-network/#Base.open) to read from or write to an external command.
```julia-repl
julia> open(`less`, "w", stdout) do io
           for i = 1:3
               println(io, i)
           end
       end
1
2
3
```
The program name and the individual arguments in a command can be accessed and iterated over as if the command were an array of strings:
```julia-repl
julia> collect(`echo "foo bar"`)
2-element Vector{String}:
 "echo"
 "foo bar"

julia> `echo "foo bar"`[2]
"foo bar"
```


## [Interpolation](#command-interpolation)
Suppose you want to do something a bit more complicated and use the name of a file in the variable `file` as an argument to a command. You can use `$` for interpolation much as you would in a string literal (see [Strings](../../devdocs/ast/#Strings)):
```julia-repl
julia> file = "/etc/passwd"
"/etc/passwd"

julia> `sort $file`
`sort /etc/passwd`
```
A common pitfall when running external programs via a shell is that if a file name contains characters that are special to the shell, they may cause undesirable behavior. Suppose, for example, rather than `/etc/passwd`, we wanted to sort the contents of the file `/Volumes/External HD/data.csv`. Let's try it:
```julia-repl
julia> file = "/Volumes/External HD/data.csv"
"/Volumes/External HD/data.csv"

julia> `sort $file`
`sort '/Volumes/External HD/data.csv'`
```
How did the file name get quoted? Julia knows that `file` is meant to be interpolated as a single argument, so it quotes the word for you. Actually, that is not quite accurate: the value of `file` is never interpreted by a shell, so there's no need for actual quoting; the quotes are inserted only for presentation to the user. This will even work if you interpolate a value as part of a shell word:
```julia-repl
julia> path = "/Volumes/External HD"
"/Volumes/External HD"

julia> name = "data"
"data"

julia> ext = "csv"
"csv"

julia> `sort $path/$name.$ext`
`sort '/Volumes/External HD/data.csv'`
```


## [Interpolation](#command-interpolation)
As you can see, the space in the `path` variable is appropriately escaped. But what if you *want* to interpolate multiple words? In that case, just use an array (or any other iterable container):
```julia-repl
julia> files = ["/etc/passwd","/Volumes/External HD/data.csv"]
2-element Vector{String}:
 "/etc/passwd"
 "/Volumes/External HD/data.csv"

julia> `grep foo $files`
`grep foo /etc/passwd '/Volumes/External HD/data.csv'`
```
If you interpolate an array as part of a shell word, Julia emulates the shell's `{a,b,c}` argument generation:
```julia-repl
julia> names = ["foo","bar","baz"]
3-element Vector{String}:
 "foo"
 "bar"
 "baz"

julia> `grep xylophone $names.txt`
`grep xylophone foo.txt bar.txt baz.txt`
```
Moreover, if you interpolate multiple arrays into the same word, the shell's Cartesian product generation behavior is emulated:
```julia-repl
julia> names = ["foo","bar","baz"]
3-element Vector{String}:
 "foo"
 "bar"
 "baz"

julia> exts = ["aux","log"]
2-element Vector{String}:
 "aux"
 "log"

julia> `rm -f $names.$exts`
`rm -f foo.aux foo.log bar.aux bar.log baz.aux baz.log`
```
Since you can interpolate literal arrays, you can use this generative functionality without needing to create temporary array objects first:
```julia-repl
julia> `rm -rf $["foo","bar","baz","qux"].$["aux","log","pdf"]`
`rm -rf foo.aux foo.log foo.pdf bar.aux bar.log bar.pdf baz.aux baz.log baz.pdf qux.aux qux.log qux.pdf`
```


## [Quoting](#Quoting)
Inevitably, one wants to write commands that aren't quite so simple, and it becomes necessary to use quotes. Here's a simple example of a Perl one-liner at a shell prompt:
```julia
sh$ perl -le '$|=1; for (0..3) { print }'
0
1
2
3
```
The Perl expression needs to be in single quotes for two reasons: so that spaces don't break the expression into multiple shell words, and so that uses of Perl variables like `$|` (yes, that's the name of a variable in Perl), don't cause interpolation. In other instances, you may want to use double quotes so that interpolation *does* occur:
```julia
sh$ first="A"
sh$ second="B"
sh$ perl -le '$|=1; print for @ARGV' "1: $first" "2: $second"
1: A
2: B
```
In general, the Julia backtick syntax is carefully designed so that you can just cut-and-paste shell commands as is into backticks and they will work: the escaping, quoting, and interpolation behaviors are the same as the shell's. The only difference is that the interpolation is integrated and aware of Julia's notion of what is a single string value, and what is a container for multiple values. Let's try the above two examples in Julia:
```julia-repl
julia> A = `perl -le '$|=1; for (0..3) { print }'`
`perl -le '$|=1; for (0..3) { print }'`

julia> run(A);
0
1
2
3

julia> first = "A"; second = "B";

julia> B = `perl -le 'print for @ARGV' "1: $first" "2: $second"`
`perl -le 'print for @ARGV' '1: A' '2: B'`

julia> run(B);
1: A
2: B
```


## [Quoting](#Quoting)
The results are identical, and Julia's interpolation behavior mimics the shell's with some improvements due to the fact that Julia supports first-class iterable objects while most shells use strings split on spaces for this, which introduces ambiguities. When trying to port shell commands to Julia, try cut and pasting first. Since Julia shows commands to you before running them, you can easily and safely just examine its interpretation without doing any damage.


## [Pipelines](#Pipelines)
Shell metacharacters, such as `|`, `&`, and `>`, need to be quoted (or escaped) inside of Julia's backticks:
```julia-repl
julia> run(`echo hello '|' sort`);
hello | sort

julia> run(`echo hello \| sort`);
hello | sort
```
This expression invokes the `echo` command with three words as arguments: `hello`, `|`, and `sort`. The result is that a single line is printed: `hello | sort`. How, then, does one construct a pipeline? Instead of using `'|'` inside of backticks, one uses [`pipeline`](../../base/base/#Base.pipeline-Tuple%7BAny,%20Any,%20Any,%20Vararg%7BAny%7D%7D):
```julia-repl
julia> run(pipeline(`echo hello`, `sort`));
hello
```
This pipes the output of the `echo` command to the `sort` command. Of course, this isn't terribly interesting since there's only one line to sort, but we can certainly do much more interesting things:
```julia-repl
julia> run(pipeline(`cut -d: -f3 /etc/passwd`, `sort -n`, `tail -n5`))
210
211
212
213
214
```
This prints the highest five user IDs on a UNIX system. The `cut`, `sort` and `tail` commands are all spawned as immediate children of the current `julia` process, with no intervening shell process. Julia itself does the work to setup pipes and connect file descriptors that is normally done by the shell. Since Julia does this itself, it retains better control and can do some things that shells cannot.
Julia can run multiple commands in parallel:
```julia-repl
julia> run(`echo hello` & `echo world`);
world
hello
```


## [Pipelines](#Pipelines)
The order of the output here is non-deterministic because the two `echo` processes are started nearly simultaneously, and race to make the first write to the [`stdout`](../../base/io-network/#Base.stdout) descriptor they share with each other and the `julia` parent process. Julia lets you pipe the output from both of these processes to another program:
```julia-repl
julia> run(pipeline(`echo world` & `echo hello`, `sort`));
hello
world
```
In terms of UNIX plumbing, what's happening here is that a single UNIX pipe object is created and written to by both `echo` processes, and the other end of the pipe is read from by the `sort` command.
IO redirection can be accomplished by passing keyword arguments `stdin`, `stdout`, and `stderr` to the `pipeline` function:
```julia
pipeline(`do_work`, stdout=pipeline(`sort`, "out.txt"), stderr="errs.txt")
```


### [Avoiding Deadlock in Pipelines](#Avoiding-Deadlock-in-Pipelines)
When reading and writing to both ends of a pipeline from a single process, it is important to avoid forcing the kernel to buffer all of the data.
For example, when reading all of the output from a command, call `read(out, String)`, not `wait(process)`, since the former will actively consume all of the data written by the process, whereas the latter will attempt to store the data in the kernel's buffers while waiting for a reader to be connected.
Another common solution is to separate the reader and writer of the pipeline into separate [`Task`](../../base/parallel/#Core.Task)s:
```julia
writer = @async write(process, "data")
reader = @async do_compute(read(process, String))
wait(writer)
fetch(reader)
```
(commonly also, reader is not a separate task, since we immediately `fetch` it anyways).


### [Complex Example](#Complex-Example)
The combination of a high-level programming language, a first-class command abstraction, and automatic setup of pipes between processes is a powerful one. To give some sense of the complex pipelines that can be created easily, here are some more sophisticated examples, with apologies for the excessive use of Perl one-liners:
```julia-repl
julia> prefixer(prefix, sleep) = `perl -nle '$|=1; print "'$prefix' ", $_; sleep '$sleep';'`;

julia> run(pipeline(`perl -le '$|=1; for(0..5){ print; sleep 1 }'`, prefixer("A",2) & prefixer("B",2)));
B 0
A 1
B 2
A 3
B 4
A 5
```
This is a classic example of a single producer feeding two concurrent consumers: one `perl` process generates lines with the numbers 0 through 5 on them, while two parallel processes consume that output, one prefixing lines with the letter "A", the other with the letter "B". Which consumer gets the first line is non-deterministic, but once that race has been won, the lines are consumed alternately by one process and then the other. (Setting `$|=1` in Perl causes each print statement to flush the [`stdout`](../../base/io-network/#Base.stdout) handle, which is necessary for this example to work. Otherwise all the output is buffered and printed to the pipe at once, to be read by just one consumer process.)
Here is an even more complex multi-stage producer-consumer example:


### [Complex Example](#Complex-Example)
```julia-repl
julia> run(pipeline(`perl -le '$|=1; for(0..5){ print; sleep 1 }'`,
           prefixer("X",3) & prefixer("Y",3) & prefixer("Z",3),
           prefixer("A",2) & prefixer("B",2)));
A X 0
B Y 1
A Z 2
B X 3
A Y 4
B Z 5
```
This example is similar to the previous one, except there are two stages of consumers, and the stages have different latency so they use a different number of parallel workers, to maintain saturated throughput.
We strongly encourage you to try all these examples to see how they work.


## [Cmd Objects](#Cmd-Objects)
The backtick syntax create an object of type [`Cmd`](../../base/base/#Base.Cmd). Such object may also be constructed directly from an existing `Cmd` or list of arguments:
```julia
run(Cmd(`pwd`, dir=".."))
run(Cmd(["pwd"], detach=true, ignorestatus=true))
```
This allows you to specify several aspects of the `Cmd`'s execution environment via keyword arguments. For example, the `dir` keyword provides control over the `Cmd`'s working directory:
```julia-repl
julia> run(Cmd(`pwd`, dir="/"));
/
```
And the `env` keyword allows you to set execution environment variables:
```julia-repl
julia> run(Cmd(`sh -c "echo foo \$HOWLONG"`, env=("HOWLONG" => "ever!",)));
foo ever!
```
See [`Cmd`](../../base/base/#Base.Cmd) for additional keyword arguments. The [`setenv`](../../base/base/#Base.setenv) and [`addenv`](../../base/base/#Base.addenv) commands provide another means for replacing or adding to the `Cmd` execution environment variables, respectively:
```julia-repl
julia> run(setenv(`sh -c "echo foo \$HOWLONG"`, ("HOWLONG" => "ever!",)));
foo ever!

julia> run(addenv(`sh -c "echo foo \$HOWLONG"`, "HOWLONG" => "ever!"));
foo ever!
```
------------------------------------------------------------------------


# Calling C and Fortran Code · The Julia Language
Source: https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/


# [Calling C and Fortran Code](#Calling-C-and-Fortran-Code)
Though most code can be written in Julia, there are many high-quality, mature libraries for numerical computing already written in C and Fortran. To allow easy use of this existing code, Julia makes it simple and efficient to call C and Fortran functions. Julia has a "no boilerplate" philosophy: functions can be called directly from Julia without any "glue" code, code generation, or compilation – even from the interactive prompt. This is accomplished just by making an appropriate call with the [`@ccall`](../../base/c/#Base.@ccall) macro (or the less convenient [`ccall`](../../base/c/#ccall) syntax, see the [`ccall` syntax section](#ccall-interface)).
The code to be called must be available as a shared library. Most C and Fortran libraries ship compiled as shared libraries already, but if you are compiling the code yourself using GCC (or Clang), you will need to use the `-shared` and `-fPIC` options. The machine instructions generated by Julia's JIT are the same as a native C call would be, so the resulting overhead is the same as calling a library function from C code. <sup>[[1\]](#footnote-1)</sup>


# [Calling C and Fortran Code](#Calling-C-and-Fortran-Code)
By default, Fortran compilers [generate mangled names](https://en.wikipedia.org/wiki/Name_mangling#Fortran) (for example, converting function names to lowercase or uppercase, often appending an underscore), and so to call a Fortran function you must pass the mangled identifier corresponding to the rule followed by your Fortran compiler. Also, when calling a Fortran function, all inputs must be passed as pointers to allocated values on the heap or stack. This applies not only to arrays and other mutable objects which are normally heap-allocated, but also to scalar values such as integers and floats which are normally stack-allocated and commonly passed in registers when using C or Julia calling conventions.
The syntax for [`@ccall`](../../base/c/#Base.@ccall) to generate a call to the library function is:
```julia
  @ccall library.function_name(argvalue1::argtype1, ...)::returntype
  @ccall function_name(argvalue1::argtype1, ...)::returntype
  @ccall $function_pointer(argvalue1::argtype1, ...)::returntype
```


# [Calling C and Fortran Code](#Calling-C-and-Fortran-Code)
where `library` is a string constant or literal (but see [Non-constant Function Specifications](#Non-constant-Function-Specifications) below). The library may be omitted, in which case the function name is resolved in the current process. This form can be used to call C library functions, functions in the Julia runtime, or functions in an application linked to Julia. The full path to the library may also be specified. Alternatively, `@ccall` may also be used to call a function pointer `$function_pointer`, such as one returned by `Libdl.dlsym`. The `argtype`s corresponds to the C-function signature and the `argvalue`s are the actual argument values to be passed to the function.
See below for how to [map C types to Julia types](#mapping-c-types-to-julia).
As a complete but simple example, the following calls the `clock` function from the standard C library on most Unix-derived systems:
```julia-repl
julia> t = @ccall clock()::Int32
2292761

julia> typeof(t)
Int32
```
`clock` takes no arguments and returns an `Int32`. To call the `getenv` function to get a pointer to the value of an environment variable, one makes a call like this:
```julia-repl
julia> path = @ccall getenv("SHELL"::Cstring)::Cstring
Cstring(@0x00007fff5fbffc45)

julia> unsafe_string(path)
"/bin/bash"
```


# [Calling C and Fortran Code](#Calling-C-and-Fortran-Code)
In practice, especially when providing reusable functionality, one generally wraps `@ccall` uses in Julia functions that set up arguments and then check for errors in whatever manner the C or Fortran function specifies. And if an error occurs it is thrown as a normal Julia exception. This is especially important since C and Fortran APIs are notoriously inconsistent about how they indicate error conditions. For example, the `getenv` C library function is wrapped in the following Julia function, which is a simplified version of the actual definition from [`env.jl`](https://github.com/JuliaLang/julia/blob/master/base/env.jl):
```julia
function getenv(var::AbstractString)
    val = @ccall getenv(var::Cstring)::Cstring
    if val == C_NULL
        error("getenv: undefined variable: ", var)
    end
    return unsafe_string(val)
end
```
The C `getenv` function indicates an error by returning `C_NULL`, but other standard C functions indicate errors in different ways, including by returning -1, 0, 1, and other special values. This wrapper throws an exception indicating the problem if the caller tries to get a non-existent environment variable:
```julia-repl
julia> getenv("SHELL")
"/bin/bash"

julia> getenv("FOOBAR")
ERROR: getenv: undefined variable: FOOBAR
```
Here is a slightly more complex example that discovers the local machine's hostname.


# [Calling C and Fortran Code](#Calling-C-and-Fortran-Code)
```julia
function gethostname()
    hostname = Vector{UInt8}(undef, 256) # MAXHOSTNAMELEN
    err = @ccall gethostname(hostname::Ptr{UInt8}, sizeof(hostname)::Csize_t)::Int32
    Base.systemerror("gethostname", err != 0)
    hostname[end] = 0 # ensure null-termination
    return GC.@preserve hostname unsafe_string(pointer(hostname))
end
```
This example first allocates an array of bytes. It then calls the C library function `gethostname` to populate the array with the hostname. Finally, it takes a pointer to the hostname buffer, and converts the pointer to a Julia string, assuming that it is a null-terminated C string.
It is common for C libraries to use this pattern of requiring the caller to allocate memory to be passed to the callee and populated. Allocation of memory from Julia like this is generally accomplished by creating an uninitialized array and passing a pointer to its data to the C function. This is why we don't use the `Cstring` type here: as the array is uninitialized, it could contain null bytes. Converting to a `Cstring` as part of the `@ccall` checks for contained null bytes and could therefore throw a conversion error.
Dereferencing `pointer(hostname)` with `unsafe_string` is an unsafe operation as it requires access to the memory allocated for `hostname` that may have been in the meanwhile garbage collected. The macro [`GC.@preserve`](../../base/base/#Base.GC.@preserve) prevents this from happening and therefore accessing an invalid memory location.


# [Calling C and Fortran Code](#Calling-C-and-Fortran-Code)
Finally, here is an example of specifying a library via a path. We create a shared library with the following content
```c
#include <stdio.h>

void say_y(int y)
{
    printf("Hello from C: got y = %d.\n", y);
}
```
and compile it with `gcc -fPIC -shared -o mylib.so mylib.c`. It can then be called by specifying the (absolute) path as the library name:
```julia-repl
julia> @ccall "./mylib.so".say_y(5::Cint)::Cvoid
Hello from C: got y = 5.
```


## [Creating C-Compatible Julia Function Pointers](#Creating-C-Compatible-Julia-Function-Pointers)
It is possible to pass Julia functions to native C functions that accept function pointer arguments. For example, to match C prototypes of the form:
```c
typedef returntype (*functiontype)(argumenttype, ...)
```
The macro [`@cfunction`](../../base/c/#Base.@cfunction) generates the C-compatible function pointer for a call to a Julia function. The arguments to [`@cfunction`](../../base/c/#Base.@cfunction) are:
1.  A Julia function
2.  The function's return type
3.  A tuple of input types, corresponding to the function signature
As with `@ccall`, the return type and the input types must be literal constants.
Currently, only the platform-default C calling convention is supported. This means that `@cfunction`-generated pointers cannot be used in calls where WINAPI expects a `stdcall` function on 32-bit Windows, but can be used on WIN64 (where `stdcall` is unified with the C calling convention).
Callback functions exposed via `@cfunction` should not throw errors, as that will return control to the Julia runtime unexpectedly and may leave the program in an undefined state.
A classic example is the standard C library `qsort` function, declared as:
```c
void qsort(void *base, size_t nitems, size_t size,
           int (*compare)(const void*, const void*));
```


## [Creating C-Compatible Julia Function Pointers](#Creating-C-Compatible-Julia-Function-Pointers)
The `base` argument is a pointer to an array of length `nitems`, with elements of `size` bytes each. `compare` is a callback function which takes pointers to two elements `a` and `b` and returns an integer less/greater than zero if `a` should appear before/after `b` (or zero if any order is permitted).
Now, suppose that we have a 1-d array `A` of values in Julia that we want to sort using the `qsort` function (rather than Julia's built-in `sort` function). Before we consider calling `qsort` and passing arguments, we need to write a comparison function:
```julia-repl
julia> function mycompare(a, b)::Cint
           return (a < b) ? -1 : ((a > b) ? +1 : 0)
       end;
```
`qsort` expects a comparison function that return a C `int`, so we annotate the return type to be `Cint`.
In order to pass this function to C, we obtain its address using the macro `@cfunction`:
```julia-repl
julia> mycompare_c = @cfunction(mycompare, Cint, (Ref{Cdouble}, Ref{Cdouble}));
```
[`@cfunction`](../../base/c/#Base.@cfunction) requires three arguments: the Julia function (`mycompare`), the return type (`Cint`), and a literal tuple of the input argument types, in this case to sort an array of `Cdouble` ([`Float64`](../../base/numbers/#Core.Float64)) elements.
The final call to `qsort` looks like this:


## [Creating C-Compatible Julia Function Pointers](#Creating-C-Compatible-Julia-Function-Pointers)
```julia-repl
julia> A = [1.3, -2.7, 4.4, 3.1];

julia> @ccall qsort(A::Ptr{Cdouble}, length(A)::Csize_t, sizeof(eltype(A))::Csize_t, mycompare_c::Ptr{Cvoid})::Cvoid

julia> A
4-element Vector{Float64}:
 -2.7
  1.3
  3.1
  4.4
```
As the example shows, the original Julia array `A` has now been sorted: `[-2.7, 1.3, 3.1, 4.4]`. Note that Julia [takes care of converting the array to a `Ptr{Cdouble}`](#automatic-type-conversion)), computing the size of the element type in bytes, and so on.
For fun, try inserting a `println("mycompare($a, $b)")` line into `mycompare`, which will allow you to see the comparisons that `qsort` is performing (and to verify that it is really calling the Julia function that you passed to it).


## [Mapping C Types to Julia](#mapping-c-types-to-julia)
It is critical to exactly match the declared C type with its declaration in Julia. Inconsistencies can cause code that works correctly on one system to fail or produce indeterminate results on a different system.
Note that no C header files are used anywhere in the process of calling C functions: you are responsible for making sure that your Julia types and call signatures accurately reflect those in the C header file.<sup>[[2\]](#footnote-2)</sup>


### [Automatic Type Conversion](#automatic-type-conversion)
Julia automatically inserts calls to the [`Base.cconvert`](../../base/c/#Base.cconvert) function to convert each argument to the specified type. For example, the following call:
```julia
@ccall "libfoo".foo(x::Int32, y::Float64)::Cvoid
```
will behave as if it were written like this:
```julia
c_x = Base.cconvert(Int32, x)
c_y = Base.cconvert(Float64, y)
GC.@preserve c_x c_y begin
    @ccall "libfoo".foo(
        Base.unsafe_convert(Int32, c_x)::Int32,
        Base.unsafe_convert(Float64, c_y)::Float64
    )::Cvoid
end
```
[`Base.cconvert`](../../base/c/#Base.cconvert) normally just calls [`convert`](../../base/base/#Base.convert), but can be defined to return an arbitrary new object more appropriate for passing to C. This should be used to perform all allocations of memory that will be accessed by the C code. For example, this is used to convert an `Array` of objects (e.g. strings) to an array of pointers.
[`Base.unsafe_convert`](../../base/c/#Base.unsafe_convert) handles conversion to [`Ptr`](../../base/c/#Core.Ptr) types. It is considered unsafe because converting an object to a native pointer can hide the object from the garbage collector, causing it to be freed prematurely.


### [Type Correspondences](#Type-Correspondences)
First, let's review some relevant Julia type terminology:
| Syntax / Keyword        | Example                                    | Description                                                                                |
|:------------------------|:-------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `mutable struct`        | `BitSet`                                   | "Leaf Type" :: A group of related data that includes a type-tag, is managed by the Julia GC, and is defined by object-identity. The type parameters of a leaf type must be fully defined (no `TypeVars` are allowed) in order for the instance to be constructed. |
| `abstract type`         | `Any`, `AbstractArray{T, N}`, `Complex{T}` | "Super Type" :: A super-type (not a leaf-type) that cannot be instantiated, but can be used to describe a group of types.                                                                                |
| `T{A}`                  | `Vector{Int}`                              | "Type Parameter" :: A specialization of a type (typically used for dispatch or storage optimization).                                                                                |
|                         |                                            | "TypeVar" :: The `T` in the type parameter declaration is referred to as a TypeVar (short for type variable).                                                                                |
| `primitive type`        | `Int`, `Float64`                           | "Primitive Type" :: A type with no fields, but a size. It is stored and defined by-value.                                                                                |
| `struct`                | `Pair{Int, Int}`                           | "Struct" :: A type with all fields defined to be constant. It is defined by-value, and may be stored with a type-tag.                                                                                |
|                         | `ComplexF64` (`isbits`)                    | "Is-Bits" :: A `primitive type`, or a `struct` type where all fields are other `isbits` types. It is defined by-value, and is stored without a type-tag.                                                                                |
| `struct ...; end`       | `nothing`                                  | "Singleton" :: a Leaf Type or Struct with no fields.                                                                                |
| `(...)` or `tuple(...)` | `(1, 2, 3)`                                | "Tuple" :: an immutable data-structure similar to an anonymous struct type, or a constant array. Represented as either an array or a struct.                                                                                |


### [Bits Types](#man-bits-types)
There are several special types to be aware of, as no other type can be defined to behave the same:
-   `Float32`
    Exactly corresponds to the `float` type in C (or `REAL*4` in Fortran).
-   `Float64`
    Exactly corresponds to the `double` type in C (or `REAL*8` in Fortran).
-   `ComplexF32`
    Exactly corresponds to the `complex float` type in C (or `COMPLEX*8` in Fortran).
-   `ComplexF64`
    Exactly corresponds to the `complex double` type in C (or `COMPLEX*16` in Fortran).
-   `Signed`
    Exactly corresponds to the `signed` type annotation in C (or any `INTEGER` type in Fortran). Any Julia type that is not a subtype of [`Signed`](../../base/numbers/#Core.Signed) is assumed to be unsigned.
<!-- -->
-   `Ref{T}`
    Behaves like a `Ptr{T}` that can manage its memory via the Julia GC.
<!-- -->
-   `Array{T,N}`
    When an array is passed to C as a `Ptr{T}` argument, it is not reinterpret-cast: Julia requires that the element type of the array matches `T`, and the address of the first element is passed.
    Therefore, if an `Array` contains data in the wrong format, it will have to be explicitly converted using a call such as `trunc.(Int32, A)`.
    To pass an array `A` as a pointer of a different type *without* converting the data beforehand (for example, to pass a `Float64` array to a function that operates on uninterpreted bytes), you can declare the argument as `Ptr{Cvoid}`.


### [Bits Types](#man-bits-types)
    If an array of eltype `Ptr{T}` is passed as a `Ptr{Ptr{T}}` argument, [`Base.cconvert`](../../base/c/#Base.cconvert) will attempt to first make a null-terminated copy of the array with each element replaced by its [`Base.cconvert`](../../base/c/#Base.cconvert) version. This allows, for example, passing an `argv` pointer array of type `Vector{String}` to an argument of type `Ptr{Ptr{Cchar}}`.
On all systems we currently support, basic C/C++ value types may be translated to Julia types as follows. Every C type also has a corresponding Julia type with the same name, prefixed by C. This can help when writing portable code (and remembering that an `int` in C is not the same as an `Int` in Julia).
**System Independent Types**


### [Bits Types](#man-bits-types)
| C name                                                  | Fortran name             | Standard Julia Alias | Julia Base Type                                                                                |
|:--------------------------------------------------------|:-------------------------|:---------------------|:------------------------------------------------------------------------------------------------------------|
| `unsigned char`                                         | `CHARACTER`              | `Cuchar`             | `UInt8`                                                                                |
| `bool` (\_Bool in C99+)                                 |                          | `Cuchar`             | `UInt8`                                                                                |
| `short`                                                 | `INTEGER*2`, `LOGICAL*2` | `Cshort`             | `Int16`                                                                                |
| `unsigned short`                                        |                          | `Cushort`            | `UInt16`                                                                                |
| `int`, `BOOL` (C, typical)                              | `INTEGER*4`, `LOGICAL*4` | `Cint`               | `Int32`                                                                                |
| `unsigned int`                                          |                          | `Cuint`              | `UInt32`                                                                                |
| `long long`                                             | `INTEGER*8`, `LOGICAL*8` | `Clonglong`          | `Int64`                                                                                |
| `unsigned long long`                                    |                          | `Culonglong`         | `UInt64`                                                                                |
| `intmax_t`                                              |                          | `Cintmax_t`          | `Int64`                                                                                |
| `uintmax_t`                                             |                          | `Cuintmax_t`         | `UInt64`                                                                                |
| `float`                                                 | `REAL*4i`                | `Cfloat`             | `Float32`                                                                                |
| `double`                                                | `REAL*8`                 | `Cdouble`            | `Float64`                                                                                |
| `complex float`                                         | `COMPLEX*8`              | `ComplexF32`         | `Complex{Float32}`                                                                                |
| `complex double`                                        | `COMPLEX*16`             | `ComplexF64`         | `Complex{Float64}`                                                                                |
| `ptrdiff_t`                                             |                          | `Cptrdiff_t`         | `Int`                                                                                |
| `ssize_t`                                               |                          | `Cssize_t`           | `Int`                                                                                |
| `size_t`                                                |                          | `Csize_t`            | `UInt`                                                                                |
| `void`                                                  |                          |                      | `Cvoid`                                                                                |
| `void` and `[[noreturn]]` or `_Noreturn`                |                          |                      | `Union{}`                                                                                |
| `void*`                                                 |                          |                      | `Ptr{Cvoid}` (or similarly `Ref{Cvoid}`)                                                                    |
| `T*` (where T represents an appropriately defined type) |                          |                      | `Ref{T}` (T may be safely mutated only if T is an isbits type)                                              |
| `char*` (or `char[]`, e.g. a string)                    | `CHARACTER*N`            |                      | `Cstring` if null-terminated, or `Ptr{UInt8}` if not                                                        |
| `char**` (or `*char[]`)                                 |                          |                      | `Ptr{Ptr{UInt8}}`                                                                                |
| `jl_value_t*` (any Julia Type)                          |                          |                      | `Any`                                                                                |
| `jl_value_t* const*` (a reference to a Julia value)     |                          |                      | `Ref{Any}` (const, since mutation would require a write barrier, which is not possible to insert correctly) |
| `va_arg`                                                |                          |                      | Not supported                                                                                |
| `...` (variadic function specification)                 |                          |                      | `T...` (where `T` is one of the above types, when using the `ccall` function)                               |
| `...` (variadic function specification)                 |                          |                      | `; va_arg1::T, va_arg2::S, etc.` (only supported with `@ccall` macro)                                       |


### [Bits Types](#man-bits-types)
The [`Cstring`](../../base/c/#Base.Cstring) type is essentially a synonym for `Ptr{UInt8}`, except the conversion to `Cstring` throws an error if the Julia string contains any embedded null characters (which would cause the string to be silently truncated if the C routine treats null as the terminator). If you are passing a `char*` to a C routine that does not assume null termination (e.g. because you pass an explicit string length), or if you know for certain that your Julia string does not contain null and want to skip the check, you can use `Ptr{UInt8}` as the argument type. `Cstring` can also be used as the [`ccall`](../../base/c/#ccall) return type, but in that case it obviously does not introduce any extra checks and is only meant to improve the readability of the call.
**System Dependent Types**
| C name          | Standard Julia Alias | Julia Base Type                              |
|:----------------|:---------------------|:---------------------------------------------|
| `char`          | `Cchar`              | `Int8` (x86, x86_64), `UInt8` (powerpc, arm) |
| `long`          | `Clong`              | `Int` (UNIX), `Int32` (Windows)              |
| `unsigned long` | `Culong`             | `UInt` (UNIX), `UInt32` (Windows)            |
| `wchar_t`       | `Cwchar_t`           | `Int32` (UNIX), `UInt16` (Windows)           |


### [Bits Types](#man-bits-types)
When calling Fortran, all inputs must be passed by pointers to heap- or stack-allocated values, so all type correspondences above should contain an additional `Ptr{..}` or `Ref{..}` wrapper around their type specification.
For string arguments (`char*`) the Julia type should be `Cstring` (if null-terminated data is expected), or either `Ptr{Cchar}` or `Ptr{UInt8}` otherwise (these two pointer types have the same effect), as described above, not `String`. Similarly, for array arguments (`T[]` or `T*`), the Julia type should again be `Ptr{T}`, not `Vector{T}`.
Julia's `Char` type is 32 bits, which is not the same as the wide-character type (`wchar_t` or `wint_t`) on all platforms.
A return type of `Union{}` means the function will not return, i.e., C++11 `[[noreturn]]` or C11 `_Noreturn` (e.g. `jl_throw` or `longjmp`). Do not use this for functions that return no value (`void`) but do return, for those, use `Cvoid` instead.
For `wchar_t*` arguments, the Julia type should be [`Cwstring`](../../base/c/#Base.Cwstring) (if the C routine expects a null-terminated string), or `Ptr{Cwchar_t}` otherwise. Note also that UTF-8 string data in Julia is internally null-terminated, so it can be passed to C functions expecting null-terminated data without making a copy (but using the `Cwstring` type will cause an error to be thrown if the string itself contains null characters).
C functions that take an argument of type `char**` can be called by using a `Ptr{Ptr{UInt8}}` type within Julia. For example, C functions of the form: