text:
# gluonnlp.model¶

GluonNLP Toolkit supplies models for common NLP tasks with pre-trained weights. By default, all requested pre-trained weights are downloaded from public repo and stored in ~/.mxnet/models/.

## Language Modeling¶

 awd_lstm_lm_1150 3-layer LSTM language model with weight-drop, variational dropout, and tied weights. awd_lstm_lm_600 3-layer LSTM language model with weight-drop, variational dropout, and tied weights. AWDRNN AWD language model by salesforce. standard_lstm_lm_200 Standard 2-layer LSTM language model with tied embedding and output weights. standard_lstm_lm_650 Standard 2-layer LSTM language model with tied embedding and output weights. standard_lstm_lm_1500 Standard 2-layer LSTM language model with tied embedding and output weights. big_rnn_lm_2048_512 Big 1-layer LSTMP language model. StandardRNN Standard RNN language model. get_model Returns a pre-defined model by name. BigRNN Big language model with LSTMP for inference.

## Convolutional Encoder¶

 ConvolutionalEncoder Convolutional encoder.

## Highway Network¶

 Highway Highway network.

## Attention Cell¶

 AttentionCell Abstract class for attention cells. MultiHeadAttentionCell Multi-head Attention Cell. MLPAttentionCell Concat the query and the key and use a single-hidden-layer MLP to get the attention score. DotProductAttentionCell Dot product attention between the query and the key.

## Sequence Sampling¶

 BeamSearchScorer Score function used in beam search. BeamSearchSampler Draw samples from the decoder by beam search. SequenceSampler Draw samples from the decoder according to the step-wise distribution.

## Other Modeling Utilities¶

 WeightDropParameter A Container holding parameters (weights) of Blocks and performs dropout. apply_weight_drop Apply weight drop to the parameter of a block. L2Normalization Normalize the input array by dividing the L2 norm along the given axis. ISDense Importance sampled Dense block, which computes sampled pred output and labels for importance sampled softmax loss during training. NCEDense Noise contrastive estimated Dense block, which computes sampled pred output and labels for noise contrastive estimation loss during training. SparseISDense Importance sampled Dense block with sparse weights, which computes sampled pred output and labels for importance sampled softmax loss during training. SparseNCEDense Noise contrastive estimated Dense block with sparse weights, which computes sampled pred output and labels for noise contrastive estimation loss during training.

## API Reference¶

Module for pre-defined NLP models.

This module contains definitions for the following model architectures: - AWD

You can construct a model with random weights by calling its constructor. Because NLP models are tied to vocabularies, you can either specify a dataset name to load and use the vocabulary of that dataset:

import gluonnlp as nlp
awd, vocab = nlp.model.awd_lstm_lm_1150(dataset_name='wikitext-2')

or directly specify a vocabulary object:

awd, vocab = nlp.model.awd_lstm_lm_1150(None, vocab=custom_vocab)

We provide pre-trained models for all the listed models. These models can constructed by passing pretrained=True:

awd, vocab = nlp.model.awd_lstm_lm_1150(dataset_name='wikitext-2'
pretrained=True)

class gluonnlp.model.AWDRNN(mode, vocab_size, embed_size, hidden_size, num_layers, tie_weights, dropout, weight_drop, drop_h, drop_i, drop_e, **kwargs)[source]

AWD language model by salesforce.

Parameters: mode (str) – The type of RNN to use. Options are ‘lstm’, ‘gru’, ‘rnn_tanh’, ‘rnn_relu’. vocab_size (int) – Size of the input vocabulary. embed_size (int) – Dimension of embedding vectors. hidden_size (int) – Number of hidden units for RNN. num_layers (int) – Number of RNN layers. tie_weights (bool, default False) – Whether to tie the weight matrices of output dense layer and input embedding layer. dropout (float) – Dropout rate to use for encoder output. weight_drop (float) – Dropout rate to use on encoder h2h weights. drop_h (float) – Dropout rate to on the output of intermediate layers of encoder. drop_i (float) – Dropout rate to on the output of embedding. drop_e (float) – Dropout rate to use on the embedding layer.
forward(inputs, begin_state=None)[source]

Implement forward computation.

Parameters: inputs (NDArray) – input tensor with shape (sequence_length, batch_size) when layout is “TNC”. begin_state (list) – initial recurrent state tensor with length equals to num_layers. the initial state with shape (1, batch_size, num_hidden) out (NDArray) – output tensor with shape (sequence_length, batch_size, input_size) when layout is “TNC”. out_states (list) – output recurrent state tensor with length equals to num_layers. the state with shape (1, batch_size, num_hidden)
class gluonnlp.model.StandardRNN(mode, vocab_size, embed_size, hidden_size, num_layers, dropout, tie_weights, **kwargs)[source]

Standard RNN language model.

Parameters: mode (str) – The type of RNN to use. Options are ‘lstm’, ‘gru’, ‘rnn_tanh’, ‘rnn_relu’. vocab_size (int) – Size of the input vocabulary. embed_size (int) – Dimension of embedding vectors. hidden_size (int) – Number of hidden units for RNN. num_layers (int) – Number of RNN layers. dropout (float) – Dropout rate to use for encoder output. tie_weights (bool, default False) – Whether to tie the weight matrices of output dense layer and input embedding layer.
forward(inputs, begin_state=None)[source]

Defines the forward computation. Arguments can be either NDArray or Symbol.

Parameters: inputs (NDArray) – input tensor with shape (sequence_length, batch_size) when layout is “TNC”. begin_state (list) – initial recurrent state tensor with length equals to num_layers-1. the initial state with shape (num_layers, batch_size, num_hidden) out (NDArray) – output tensor with shape (sequence_length, batch_size, input_size) when layout is “TNC”. out_states (list) – output recurrent state tensor with length equals to num_layers-1. the state with shape (num_layers, batch_size, num_hidden)
class gluonnlp.model.BigRNN(vocab_size, embed_size, hidden_size, num_layers, projection_size, embed_dropout=0.0, encode_dropout=0.0, **kwargs)[source]

Big language model with LSTMP for inference.

Parameters: vocab_size (int) – Size of the input vocabulary. embed_size (int) – Dimension of embedding vectors. hidden_size (int) – Number of hidden units for LSTMP. num_layers (int) – Number of LSTMP layers. projection_size (int) – Number of projection units for LSTMP. embed_dropout (float) – Dropout rate to use for embedding output. encode_dropout (float) – Dropout rate to use for encoder output.
forward(inputs, begin_state)[source]

Implement forward computation.

Parameters: inputs (NDArray) – input tensor with shape (sequence_length, batch_size) when layout is “TNC”. begin_state (list) – initial recurrent state tensor with length equals to num_layers*2. For each layer the two initial states have shape (batch_size, num_hidden) and (batch_size, num_projection) out (NDArray) – output tensor with shape (sequence_length, batch_size, vocab_size) when layout is “TNC”. out_states (list) – output recurrent state tensor with length equals to num_layers*2. For each layer the two initial states have shape (batch_size, num_hidden) and (batch_size, num_projection)
gluonnlp.model.awd_lstm_lm_1150(dataset_name=None, vocab=None, pretrained=False, ctx=cpu(0), root='~/.mxnet/models', **kwargs)[source]

3-layer LSTM language model with weight-drop, variational dropout, and tied weights.

Embedding size is 400, and hidden layer size is 1150.

Parameters: dataset_name (str or None, default None) – The dataset name on which the pre-trained model is trained. Options are ‘wikitext-2’. If specified, then the returned vocabulary is extracted from the training set of the dataset. If None, then vocab is required, for specifying embedding weight size, and is directly returned. The pre-trained model achieves 73.32/69.74 ppl on Val and Test of wikitext-2 respectively. vocab (gluonnlp.Vocab or None, default None) – Vocab object to be used with the language model. Required when dataset_name is not specified. pretrained (bool, default False) – Whether to load the pre-trained weights for model. ctx (Context, default CPU) – The context in which to load the pre-trained weights. root (str, default '~/.mxnet/models') – Location for keeping the model parameters. gluon.Block, gluonnlp.Vocab
gluonnlp.model.awd_lstm_lm_600(dataset_name=None, vocab=None, pretrained=False, ctx=cpu(0), root='~/.mxnet/models', **kwargs)[source]

3-layer LSTM language model with weight-drop, variational dropout, and tied weights.

Embedding size is 200, and hidden layer size is 600.

Parameters: dataset_name (str or None, default None) – The dataset name on which the pre-trained model is trained. Options are ‘wikitext-2’. If specified, then the returned vocabulary is extracted from the training set of the dataset. If None, then vocab is required, for specifying embedding weight size, and is directly returned. The pre-trained model achieves 84.61/80.96 ppl on Val and Test of wikitext-2 respectively. vocab (gluonnlp.Vocab or None, default None) – Vocab object to be used with the language model. Required when dataset_name is not specified. pretrained (bool, default False) – Whether to load the pre-trained weights for model. ctx (Context, default CPU) – The context in which to load the pre-trained weights. root (str, default '~/.mxnet/models') – Location for keeping the model parameters. gluon.Block, gluonnlp.Vocab
gluonnlp.model.standard_lstm_lm_200(dataset_name=None, vocab=None, pretrained=False, ctx=cpu(0), root='~/.mxnet/models', **kwargs)[source]

Standard 2-layer LSTM language model with tied embedding and output weights.

Both embedding and hidden dimensions are 200.

Parameters: dataset_name (str or None, default None) – The dataset name on which the pre-trained model is trained. Options are ‘wikitext-2’. If specified, then the returned vocabulary is extracted from the training set of the dataset. If None, then vocab is required, for specifying embedding weight size, and is directly returned. The pre-trained model achieves 108.25/102.26 ppl on Val and Test of wikitext-2 respectively. vocab (gluonnlp.Vocab or None, default None) – Vocabulary object to be used with the language model. Required when dataset_name is not specified. pretrained (bool, default False) – Whether to load the pre-trained weights for model. ctx (Context, default CPU) – The context in which to load the pre-trained weights. root (str, default '~/.mxnet/models') – Location for keeping the model parameters. gluon.Block, gluonnlp.Vocab
gluonnlp.model.standard_lstm_lm_650(dataset_name=None, vocab=None, pretrained=False, ctx=cpu(0), root='~/.mxnet/models', **kwargs)[source]

Standard 2-layer LSTM language model with tied embedding and output weights.

Both embedding and hidden dimensions are 650.

Parameters: dataset_name (str or None, default None) – The dataset name on which the pre-trained model is trained. Options are ‘wikitext-2’. If specified, then the returned vocabulary is extracted from the training set of the dataset. If None, then vocab is required, for specifying embedding weight size, and is directly returned. The pre-trained model achieves 98.96/93.90 ppl on Val and Test of wikitext-2 respectively. vocab (gluonnlp.Vocab or None, default None) – Vocabulary object to be used with the language model. Required when dataset_name is not specified. pretrained (bool, default False) – Whether to load the pre-trained weights for model. ctx (Context, default CPU) – The context in which to load the pre-trained weights. root (str, default '~/.mxnet/models') – Location for keeping the model parameters. gluon.Block, gluonnlp.Vocab
gluonnlp.model.standard_lstm_lm_1500(dataset_name=None, vocab=None, pretrained=False, ctx=cpu(0), root='~/.mxnet/models', **kwargs)[source]

Standard 2-layer LSTM language model with tied embedding and output weights.

Both embedding and hidden dimensions are 1500.

Parameters: dataset_name (str or None, default None) – The dataset name on which the pre-trained model is trained. Options are ‘wikitext-2’. If specified, then the returned vocabulary is extracted from the training set of the dataset. If None, then vocab is required, for specifying embedding weight size, and is directly returned. The pre-trained model achieves 98.29/92.83 ppl on Val and Test of wikitext-2 respectively. vocab (gluonnlp.Vocab or None, default None) – Vocabulary object to be used with the language model. Required when dataset_name is not specified. pretrained (bool, default False) – Whether to load the pre-trained weights for model. ctx (Context, default CPU) – The context in which to load the pre-trained weights. root (str, default '~/.mxnet/models') – Location for keeping the model parameters. gluon.Block, gluonnlp.Vocab
gluonnlp.model.big_rnn_lm_2048_512(dataset_name=None, vocab=None, pretrained=False, ctx=cpu(0), root='~/.mxnet/models', **kwargs)[source]

Big 1-layer LSTMP language model.

Both embedding and projection size are 512. Hidden size is 2048.

Parameters: dataset_name (str or None, default None) – The dataset name on which the pre-trained model is trained. Options are ‘gbw’. If specified, then the returned vocabulary is extracted from the training set of the dataset. If None, then vocab is required, for specifying embedding weight size, and is directly returned. The pre-trained model achieves 44.05 ppl on Test of GBW dataset. vocab (gluonnlp.Vocab or None, default None) – Vocabulary object to be used with the language model. Required when dataset_name is not specified. pretrained (bool, default False) – Whether to load the pre-trained weights for model. ctx (Context, default CPU) – The context in which to load the pre-trained weights. root (str, default '~/.mxnet/models') – Location for keeping the model parameters. gluon.Block, gluonnlp.Vocab
class gluonnlp.model.BeamSearchScorer(alpha=1.0, K=5.0, from_logits=True, **kwargs)[source]

Score function used in beam search.

Implements the length-penalized score function used in the GNMT paper:

scores = (log_probs + scores) / length_penalty
length_penalty = (K + length)^\alpha / (K + 1)^\alpha

Parameters: alpha (float, default 1.0) – K (float, default 5.0) – from_logits (bool, default True) – Whether input is a log probability (usually from log_softmax) instead of unnormalized numbers.
hybrid_forward(F, outputs, scores, step)[source]

Overrides to construct symbolic graph for this Block.

Parameters: x (Symbol or NDArray) – The first input tensor. *args (list of Symbol or list of NDArray) – Additional input tensors.
class gluonnlp.model.BeamSearchSampler(beam_size, decoder, eos_id, scorer=BeamSearchScorer( ), max_length=100)[source]

Draw samples from the decoder by beam search.

Parameters: beam_size (int) – The beam size. decoder (callable) – Function of the one-step-ahead decoder, should have the form: outputs, new_states = decoder(step_input, states)  The outputs, input should follow these rules: step_input has shape (batch_size,), outputs has shape (batch_size, V), states and new_states have the same structure and the leading dimension of the inner NDArrays is the batch dimension. eos_id (int) – Id of the EOS token. No other elements will be appended to the sample if it reaches eos_id. scorer (BeamSearchScorer, default BeamSearchScorer(alpha=1.0, K=5)) – The score function used in beam search. max_length (int, default 100) – The maximum search length.
class gluonnlp.model.HybridBeamSearchSampler(batch_size, beam_size, decoder, eos_id, scorer=BeamSearchScorer( ), max_length=100, vocab_size=None, prefix=None, params=None)[source]

Draw samples from the decoder by beam search.

Parameters: batch_size (int) – The batch size. beam_size (int) – The beam size. decoder (callable, must be hybridizable) – Function of the one-step-ahead decoder, should have the form: outputs, new_states = decoder(step_input, states)  The outputs, input should follow these rules: step_input has shape (batch_size,), outputs has shape (batch_size, V), states and new_states have the same structure and the leading dimension of the inner NDArrays is the batch dimension. eos_id (int) – Id of the EOS token. No other elements will be appended to the sample if it reaches eos_id. scorer (BeamSearchScorer, default BeamSearchScorer(alpha=1.0, K=5), must be hybridizable) – The score function used in beam search. max_length (int, default 100) – The maximum search length. vocab_size (int, default None, meaning decoder._vocab_size) – The vocabulary size
hybrid_forward(F, inputs, states)[source]

Sample by beam search.

Parameters: F – inputs (NDArray or Symbol) – The initial input of the decoder. Shape is (batch_size,). states (Object that contains NDArrays or Symbols) – The initial states of the decoder. samples (NDArray or Symbol) – Samples draw by beam search. Shape (batch_size, beam_size, length). dtype is int32. scores (NDArray or Symbol) – Scores of the samples. Shape (batch_size, beam_size). We make sure that scores[i, :] are in descending order. valid_length (NDArray or Symbol) – The valid length of the samples. Shape (batch_size, beam_size). dtype will be int32.
class gluonnlp.model.SequenceSampler(beam_size, decoder, eos_id, max_length=100, temperature=1.0)[source]

Draw samples from the decoder according to the step-wise distribution.

Parameters: beam_size (int) – The beam size. decoder (callable) – Function of the one-step-ahead decoder, should have the form: outputs, new_states = decoder(step_input, states)  The outputs, input should follow these rules: step_input has shape (batch_size,) outputs is the unnormalized prediction before softmax with shape (batch_size, V) states and new_states have the same structure and the leading dimension of the inner NDArrays is the batch dimension. eos_id (int) – Id of the EOS token. No other elements will be appended to the sample if it reaches eos_id. max_length (int, default 100) – The maximum search length. temperature (float, default 1.0) – Softmax temperature.
class gluonnlp.model.AttentionCell(prefix=None, params=None)[source]

Abstract class for attention cells. Extend the class to implement your own attention method. One typical usage is to define your own _compute_weight() function to calculate the weights:

cell = AttentionCell()
out = cell(query, key, value, mask)

forward(query, key, value=None, mask=None)[source]

Defines the forward computation. Arguments can be either NDArray or Symbol.

hybrid_forward(F, query, key, value, mask=None)[source]

Overrides to construct symbolic graph for this Block.

Parameters: x (Symbol or NDArray) – The first input tensor. *args (list of Symbol or list of NDArray) – Additional input tensors.
class gluonnlp.model.MultiHeadAttentionCell(base_cell, query_units, key_units, value_units, num_heads, use_bias=True, weight_initializer=None, bias_initializer='zeros', prefix=None, params=None)[source]

In the MultiHeadAttentionCell, the input query/key/value will be linearly projected for num_heads times with different projection matrices. Each projected key, value, query will be used to calculate the attention weights and values. The output of each head will be concatenated to form the final output.

The idea is first proposed in “[Arxiv2014] Neural Turing Machines” and is later adopted in “[NIPS2017] Attention is All You Need” to solve the Neural Machine Translation problem.

Parameters: base_cell (AttentionCell) – query_units (int) – Total number of projected units for query. Must be divided exactly by num_heads. key_units (int) – Total number of projected units for key. Must be divided exactly by num_heads. value_units (int) – Total number of projected units for value. Must be divided exactly by num_heads. num_heads (int) – Number of parallel attention heads use_bias (bool, default True) – Whether to use bias when projecting the query/key/values weight_initializer (str or Initializer or None, default None) – Initializer of the weights. bias_initializer (str or Initializer, default ‘zeros’) – Initializer of the bias. prefix (str or None, default None) – See document of Block. params (str or None, default None) – See document of Block.
class gluonnlp.model.MLPAttentionCell(units, act=Activation(tanh), normalized=False, dropout=0.0, weight_initializer=None, bias_initializer='zeros', prefix=None, params=None)[source]

Concat the query and the key and use a single-hidden-layer MLP to get the attention score. We provide two mode, the standard mode and the normalized mode.

In the standard mode:

score = v tanh(W [h_q, h_k] + b)

In the normalized mode (Same as TensorFlow):

score = g v / ||v||_2 tanh(W [h_q, h_k] + b)

This type of attention is first proposed in

Parameters: units (int) – act (Activation, default nn.Activation('tanh')) – normalized (bool, default False) – Whether to normalize the weight that maps the embedded hidden states to the final score. This strategy can be interpreted as a type of “[NIPS2016] Weight Normalization”. dropout (float, default 0.0) – Attention dropout. weight_initializer (str or Initializer or None, default None) – Initializer of the weights. bias_initializer (str or Initializer, default ‘zeros’) – Initializer of the bias. prefix (str or None, default None) – See document of Block. params (ParameterDict or None, default None) – See document of Block.
class gluonnlp.model.DotProductAttentionCell(units=None, luong_style=False, scaled=True, normalized=False, use_bias=True, dropout=0.0, weight_initializer=None, bias_initializer='zeros', prefix=None, params=None)[source]

Dot product attention between the query and the key.

Depending on parameters, defined as:

units is None:
score = <h_q, h_k>
units is not None and luong_style is False:
score = <W_q h_q, W_k h_k>
units is not None and luong_style is True:
score = <W h_q, h_k>

Parameters: units (int or None, default None) – Project the query and key to vectors with units dimension before applying the attention. If set to None, the query vector and the key vector are directly used to compute the attention and should have the same dimension: If the units is None, score = Else if the units is not None and luong_style is False: score = Else if the units is not None and luong_style is True: score =  luong_style (bool, default False) – If turned on, the score will be: score =  units must be the same as the dimension of the key vector scaled (bool, default True) – Whether to divide the attention weights by the sqrt of the query dimension. This is first proposed in “[NIPS2017] Attention is all you need.”: score = / sqrt(dim_q)  normalized (bool, default False) – If turned on, the cosine distance is used, i.e: score =  use_bias (bool, default True) – Whether to use bias in the projection layers. dropout (float, default 0.0) – Attention dropout weight_initializer (str or Initializer or None, default None) – Initializer of the weights bias_initializer (str or Initializer, default ‘zeros’) – Initializer of the bias prefix (str or None, default None) – See document of Block. params (str or None, default None) – See document of Block.
gluonnlp.model.apply_weight_drop(block, local_param_regex, rate, axes=(), weight_dropout_mode='training')[source]

Apply weight drop to the parameter of a block.

Parameters: block (Block or HybridBlock) – The block whose parameter is to be applied weight-drop. local_param_regex (str) – The regex for parameter names used in the self.params.get(), such as ‘weight’. rate (float) – Fraction of the input units to drop. Must be a number between 0 and 1. axes (tuple of int, default ()) – The axes on which dropout mask is shared. If empty, regular dropout is applied. weight_drop_mode ({'training', 'always'}, default 'training') – Whether the weight dropout should be applied only at training time, or always be applied.

Examples

>>> import mxnet as mx
>>> from mxnet import gluon
>>> import gluonnlp as nlp
>>> net = gluon.rnn.LSTM(10, num_layers=2, bidirectional=True)
>>> nlp.model.apply_weight_drop(net, r'.*h2h_weight\Z', 0.5)
>>> net.collect_params()
lstm0_ (
Parameter lstm0_l0_i2h_weight (shape=(40, 0), dtype=<class 'numpy.float32'>)
WeightDropParameter lstm0_l0_h2h_weight
(shape=(40, 10), dtype=<class 'numpy.float32'>, rate=0.5, mode=training)
Parameter lstm0_l0_i2h_bias (shape=(40,), dtype=<class 'numpy.float32'>)
Parameter lstm0_l0_h2h_bias (shape=(40,), dtype=<class 'numpy.float32'>)
Parameter lstm0_r0_i2h_weight (shape=(40, 0), dtype=<class 'numpy.float32'>)
WeightDropParameter lstm0_r0_h2h_weight
(shape=(40, 10), dtype=<class 'numpy.float32'>, rate=0.5, mode=training)
Parameter lstm0_r0_i2h_bias (shape=(40,), dtype=<class 'numpy.float32'>)
Parameter lstm0_r0_h2h_bias (shape=(40,), dtype=<class 'numpy.float32'>)
Parameter lstm0_l1_i2h_weight (shape=(40, 20), dtype=<class 'numpy.float32'>)
WeightDropParameter lstm0_l1_h2h_weight
(shape=(40, 10), dtype=<class 'numpy.float32'>, rate=0.5, mode=training)
Parameter lstm0_l1_i2h_bias (shape=(40,), dtype=<class 'numpy.float32'>)
Parameter lstm0_l1_h2h_bias (shape=(40,), dtype=<class 'numpy.float32'>)
Parameter lstm0_r1_i2h_weight (shape=(40, 20), dtype=<class 'numpy.float32'>)
WeightDropParameter lstm0_r1_h2h_weight
(shape=(40, 10), dtype=<class 'numpy.float32'>, rate=0.5, mode=training)
Parameter lstm0_r1_i2h_bias (shape=(40,), dtype=<class 'numpy.float32'>)
Parameter lstm0_r1_h2h_bias (shape=(40,), dtype=<class 'numpy.float32'>)
)
>>> net.initialize()
...     print(net(mx.nd.ones((3, 4, 5))).max())
[0.00488924]
<NDArray 1 @cpu(0)>
...     print(net(mx.nd.ones((3, 4, 5))).max())
[0.00475577]
<NDArray 1 @cpu(0)>

class gluonnlp.model.WeightDropParameter(parameter, rate=0.0, mode='training', axes=())[source]

A Container holding parameters (weights) of Blocks and performs dropout.

Parameters: parameter (Parameter) – The parameter which drops out. rate (float, default 0.0) – Fraction of the input units to drop. Must be a number between 0 and 1. Dropout is not applied if dropout_rate is 0. mode (str, default 'training') – Whether to only turn on dropout during training or to also turn on for inference. Options are ‘training’ and ‘always’. axes (tuple of int, default ()) – Axes on which dropout mask is shared.
data(ctx=None)[source]

Returns a copy of this parameter on one context. Must have been initialized on this context before.

Parameters: ctx (Context) – Desired context. NDArray on ctx
class gluonnlp.model.RNNCellLayer(rnn_cell, layout='TNC', **kwargs)[source]

A block that takes an rnn cell and makes it act like rnn layer.

Parameters: rnn_cell (Cell) – The cell to wrap into a layer-like block. layout (str, default 'TNC') – The output layout of the layer.
forward(inputs, states=None)[source]

Defines the forward computation. Arguments can be either NDArray or Symbol.

class gluonnlp.model.L2Normalization(axis=-1, eps=1e-06, **kwargs)[source]

Normalize the input array by dividing the L2 norm along the given axis.

..code

out = data / (sqrt(sum(data**2, axis)) + eps)
Parameters: axis (int, default -1) – The axis to compute the norm value. eps (float, default 1E-6) – The epsilon value to avoid dividing zero
hybrid_forward(F, x)[source]

Overrides to construct symbolic graph for this Block.

Parameters: x (Symbol or NDArray) – The first input tensor. *args (list of Symbol or list of NDArray) – Additional input tensors.
class gluonnlp.model.Highway(input_size, num_layers, activation='relu', highway_bias=<gluonnlp.initializer.initializer.HighwayBias object>, **kwargs)[source]

Highway network.

We implemented the highway network proposed in the following work:

@article{srivastava2015highway,
title={Highway networks},
author={Srivastava, Rupesh Kumar and Greff, Klaus and Schmidhuber, J{\"u}rgen},
journal={arXiv preprint arXiv:1505.00387},
year={2015}
}

The full version of the work:

@inproceedings{srivastava2015training,
title={Training very deep networks},
author={Srivastava, Rupesh K and Greff, Klaus and Schmidhuber, J{\"u}rgen},
booktitle={Advances in neural information processing systems},
pages={2377--2385},
year={2015}
}

A Highway layer is defined as below:

$y = (1 - t) * x + t * f(A(x))$

which is a gated combination of a linear transform and a non-linear transform of its input, where $$x$$ is the input tensor, $$A$$ is a linear transformer, $$f$$ is an element-wise non-linear transformer, and $$t$$ is an element-wise transform gate, and $$1-t$$ refers to carry gate.

Parameters: input_size (int) – The dimension of the input tensor. We assume the input has shape (batch_size, input_size). num_layers (int) – The number of highway layers to apply to the input. activation (str, default 'relu') – The non-linear activation function to use. If you don’t specify anything, no activation is applied (ie. “linear” activation: a(x) = x). highway_bias (HighwayBias,) – default HighwayBias(nonlinear_transform_bias=0.0, transform_gate_bias=-2.0) The biases applied to the highway layer. We set the default according to the above original work.
hybrid_forward(F, inputs, **kwargs)[source]

Forward computation for highway layer

Parameters: inputs (NDArray) – The input tensor is of shape (batch_size, input_size). outputs – The output tensor is of the same shape with input tensor (batch_size, input_size). NDArray
class gluonnlp.model.ConvolutionalEncoder(embed_size=15, num_filters=(25, 50, 75, 100, 125, 150), ngram_filter_sizes=(1, 2, 3, 4, 5, 6), conv_layer_activation='tanh', num_highway=1, highway_layer_activation='relu', highway_bias=<gluonnlp.initializer.initializer.HighwayBias object>, output_size=None, **kwargs)[source]

Convolutional encoder.

We implement the convolutional encoder proposed in the following work:

@inproceedings{kim2016character,
title={Character-Aware Neural Language Models.},
author={Kim, Yoon and Jernite, Yacine and Sontag, David and Rush, Alexander M},
booktitle={AAAI},
pages={2741--2749},
year={2016}
}

Parameters: embed_size (int, default 15) – The input dimension to the encoder. We set the default according to the original work’s experiments on PTB dataset with Char-small model setting. num_filters (Tuple[int], default (25, 50, 75, 100, 125, 150)) – The output dimension for each convolutional layer according to the filter sizes, which are the number of the filters learned by the layers. We set the default according to the original work’s experiments on PTB dataset with Char-small model setting. ngram_filter_sizes (Tuple[int], default (1, 2, 3, 4, 5, 6)) – The size of each convolutional layer, and len(ngram_filter_sizes) equals to the number of convolutional layers. We set the default according to the original work’s experiments on PTB dataset with Char-small model setting. conv_layer_activation (str, default 'tanh') – Activation function to be used after convolutional layer. We set the default according to the original work’s experiments on PTB dataset with Char-small model setting. num_highway (int, default '1') – The number of layers of the Highway layer. We set the default according to the original work’s experiments on PTB dataset with Char-small model setting. highway_layer_activation (str, default 'relu') – Activation function to be used after highway layer. If you don’t specify anything, no activation is applied (ie. “linear” activation: a(x) = x). We set the default according to the original work’s experiments on PTB dataset with Char-small model setting. highway_bias (HighwayBias,) – default HighwayBias(nonlinear_transform_bias=0.0, transform_gate_bias=-2.0) The biases applied to the highway layer. We set the default according to the above original work. output_size (int, default None) – The output dimension after conducting the convolutions and max pooling, and applying highways, as well as linear projection.
hybrid_forward(F, inputs, mask=None)[source]

Forward computation for char_encoder

Parameters: inputs (NDArray) – The input tensor is of shape (seq_len, batch_size, embedding_size) TNC. mask (NDArray) – The mask applied to the input of shape (seq_len, batch_size), the mask will be broadcasted along the embedding dimension. output – The output of the encoder with shape (batch_size, output_Size) NDArray
class gluonnlp.model.ISDense(num_classes, num_sampled, in_unit, remove_accidental_hits=True, dtype='float32', weight_initializer=None, bias_initializer='zeros', sparse_grad=True, prefix=None, params=None)[source]

Importance sampled Dense block, which computes sampled pred output and labels for importance sampled softmax loss during training.

Please use loss.SoftmaxCrossEntropyLoss for sampled softmax loss.

Note

If sparse_grad is set to True, the gradient w.r.t input and output embeddings will be sparse. Only a subset of optimizers support sparse gradients, including SGD, AdaGrad and Adam. By default lazy_update is turned on for these optimizers, which may perform differently from standard updates. For more details, please check the Optimization API at https://mxnet.incubator.apache.org/api/python/optimization/optimization.html

Example:

# network with importance sampling for training
encoder = Encoder(..)
decoder = ISDense(..)
loss = SoftmaxCrossEntropyLoss()

# training
for x, y, sampled_values in train_batches:
pred, new_targets = train_net(x, sampled_values, y)
l = loss(pred, new_targets)

# network for testing

# testing
for x, y in test_batches:
pred = test_net(x)
l = loss(pred, y)

Parameters: num_classes (int) – Number of possible classes. num_sampled (int) – Number of classes randomly sampled for each batch. in_unit (int) – Dimensionality of the input space. remove_accidental_hits (bool, default True) – Whether to remove “accidental hits” when a sampled candidate is equal to one of the true classes. dtype (str or np.dtype, default 'float32') – Data type of output embeddings. weight_initializer (str or Initializer, optional) – Initializer for the kernel weights matrix. bias_initializer (str or Initializer, optional) – Initializer for the bias vector. sparse_grad (bool, default True.) – Whether to use sparse gradient. Inputs – x: A tensor of shape (batch_size, in_unit). The forward activation of the input network. sampled_values : A list of three tensors for sampled_classes with shape (num_samples,), expected_count_sampled with shape (num_samples,), and expected_count_true with shape (sequence_length, batch_size). label: A tensor of shape (batch_size,1). The target classes. Outputs – out: A tensor of shape (batch_size, 1+num_sampled). The output probability for the true class and sampled classes new_targets: A tensor of shape (batch_size,). The new target classes.
class gluonnlp.model.NCEDense(num_classes, num_sampled, in_unit, remove_accidental_hits=False, dtype='float32', weight_initializer=None, bias_initializer='zeros', sparse_grad=True, prefix=None, params=None)[source]

Noise contrastive estimated Dense block, which computes sampled pred output and labels for noise contrastive estimation loss during training.

Please use loss.SigmoidBinaryCrossEntropyLoss for noise contrastive estimation loss during training.

Note

If sparse_grad is set to True, the gradient w.r.t input and output embeddings will be sparse. Only a subset of optimizers support sparse gradients, including SGD, AdaGrad and Adam. By default lazy_update is turned on for these optimizers, which may perform differently from standard updates. For more details, please check the Optimization API at: https://mxnet.incubator.apache.org/api/python/optimization/optimization.html

Example:

# network with sampling for training
encoder = Encoder(..)
decoder = NCEDense(..)
loss_train = SigmoidBinaryCrossEntropyLoss()

# training
for x, y, sampled_values in train_batches:
pred, new_targets = train_net(x, sampled_values, y)
l = loss_train(pred, new_targets)

# network for testing
loss_test = SoftmaxCrossEntropyLoss()

# testing
for x, y in test_batches:
pred = test_net(x)
l = loss_test(pred, y)

Parameters: num_classes (int) – Number of possible classes. num_sampled (int) – Number of classes randomly sampled for each batch. in_unit (int) – Dimensionality of the input space. remove_accidental_hits (bool, default False) – Whether to remove “accidental hits” when a sampled candidate is equal to one of the true classes. dtype (str or np.dtype, default 'float32') – Data type of output embeddings. weight_initializer (str or Initializer, optional) – Initializer for the kernel weights matrix. bias_initializer (str or Initializer, optional) – Initializer for the bias vector. sparse_grad (bool, default True.) – Whether to use sparse gradient. Inputs – x: A tensor of shape (batch_size, in_unit). The forward activation of the input network. sampled_values : A list of three tensors for sampled_classes with shape (num_samples,), expected_count_sampled with shape (num_samples,), and expected_count_true with shape (sequence_length, batch_size). label: A tensor of shape (batch_size,1). The target classes. Outputs – out: A tensor of shape (batch_size, 1+num_sampled). The output probability for the true class and sampled classes new_targets: A tensor of shape (batch_size, 1+num_sampled). The new target classes.
class gluonnlp.model.SparseISDense(num_classes, num_sampled, in_unit, remove_accidental_hits=True, dtype='float32', weight_initializer=None, bias_initializer='zeros', prefix=None, params=None)[source]

Importance sampled Dense block with sparse weights, which computes sampled pred output and labels for importance sampled softmax loss during training.

Please use loss.SoftmaxCrossEntropyLoss for sampled softmax loss.

The block is designed for distributed training with extremely large number of classes to reduce communication overhead and memory consumption. Both weight and gradient w.r.t. weight are RowSparseNDArray.

Note

Different from ISDense block, the weight parameter is stored in row_sparse format, which helps reduce memory consumption and communication overhead during multi-GPU training. However, sparse parameters cannot be shared with other blocks, nor could we hybridize a block containinng sparse parameters. Therefore, the parameters have to be saved before they are used for testing.

Example:

# network with importance sampled softmax for training
encoder = Encoder(..)
loss = SoftmaxCrossEntropyLoss()

# training
for x, y, sampled_values in train_batches:
pred, new_targets = train_net(x, sampled_values, y)
l = loss(pred, new_targets)

# save params
train_net.save_parameters('net.params')

# network for testing

# testing
for x, y in test_batches:
pred = test_net(x)
l = loss(pred, y)

Parameters: num_classes (int) – Number of possible classes. num_sampled (int) – Number of classes randomly sampled for each batch. in_unit (int) – Dimensionality of the input space. remove_accidental_hits (bool, default True) – Whether to remove “accidental hits” when a sampled candidate is equal to one of the true classes. dtype (str or np.dtype, default 'float32') – Data type of output embeddings. weight_initializer (str or Initializer, optional) – Initializer for the kernel weights matrix. bias_initializer (str or Initializer, optional) – Initializer for the bias vector. Inputs – x: A tensor of shape (batch_size, in_unit). The forward activation of the input network. sampled_values : A list of three tensors for sampled_classes with shape (num_samples,), expected_count_sampled with shape (num_samples,), and expected_count_true with shape (sequence_length, batch_size). label: A tensor of shape (batch_size,1). The target classes. Outputs – out: A tensor of shape (batch_size, 1+num_sampled). The output probability for the true class and sampled classes new_targets: A tensor of shape (batch_size,). The new target classes.
class gluonnlp.model.SparseNCEDense(num_classes, num_sampled, in_unit, remove_accidental_hits=True, dtype='float32', weight_initializer=None, bias_initializer='zeros', prefix=None, params=None)[source]

Noise contrastive estimated Dense block with sparse weights, which computes sampled pred output and labels for noise contrastive estimation loss during training.

Please use loss.SigmoidBinaryCrossEntropyLoss for noise contrastive estimation loss during training.

The block is designed for distributed training with extremely large number of classes to reduce communication overhead and memory consumption. Both weight and gradient w.r.t. weight are RowSparseNDArray.

Note

Different from NCEDense block, the weight parameter is stored in row_sparse format, which helps reduce memory consumption and communication overhead during multi-GPU training. However, sparse parameters cannot be shared with other blocks, nor could we hybridize a block containinng sparse parameters. Therefore, the parameters have to be saved before they are used for testing.

Example:

# network with importance sampled softmax for training
encoder = Encoder(..)
train_loss = SigmoidBinaryCrossEntropyLoss()

# training
for x, y, sampled_values in train_batches:
pred, new_targets = train_net(x, sampled_values, y)
l = train_loss(pred, new_targets)

# save params
train_net.save_parameters('net.params')

# network for testing

gluonnlp.model.get_model(name, dataset_name='wikitext-2', **kwargs)[source]


text:
# Buck Converter inductor sizing question

Discussion in 'Homework Help' started by notoriusjt2, Oct 18, 2010.

1. ### notoriusjt2 Thread Starter Member

Feb 4, 2010
209
0

my work

through trial and error I chose 22.5microH to work with

D=duty ratio is the fraction of the switching period that the switch is closed

I think this statement may be throwing me off
"Determine the size of the inductor such that the minimum inductor current is 25% of the average inductor current."

so does that mean that if the average inductor current is 1.25A then the minimum inductor current has to be 25% of that? or .3125A

I have tried all these values of inductors and none of them result in that answer

2. ### t_n_k AAC Fanatic!

Mar 6, 2009
5,448
782
You have the mean current of 1.25A.

The minimum ripple value is to be 25% of that or 0.3125A.

The difference between 1.25A and 0.3125A is 0.9375A.

The peak-to-peak ripple current is then 2*0.9375 = 1.875A

ΔIl=1.875={[Vs-Vo]/L}*DT

DT=0.6666/40000=16.67 us

L= [Vs-Vo]*DT/ΔIl=[30-20]*16.67/1.875 uH = 88.9 uH

3. ### notoriusjt2 Thread Starter Member

Feb 4, 2010
209
0
so$\Delta$Il is the peak to peak ripple current??

in my textbook it defines it as the "change in current"

4. ### t_n_k AAC Fanatic!

Mar 6, 2009
5,448
782
Yes - my wording is intended to imply the same thing.

BTW - Why did you delete the other post - it might have been of interest to others?

Last edited: Oct 18, 2010
5. ### notoriusjt2 Thread Starter Member

Feb 4, 2010
209
0
I didn't delete it... It just disappeared. I still need to work out that problem so I guess I'll repost it. Could a mod have deleted it?

6. ### t_n_k AAC Fanatic!

Mar 6, 2009
5,448
782
That's strange. There was nothing I recollect in the content that would make it a target for deletion - even something as trivial as duplication of an existing thread. Might be worth informing the administrator.

Apr 5, 2008
15,641
2,344
Hello,

I made a remark on the missing posts to the other forum leaders.

Bertus

8. ### hgmjr Moderator

Jan 28, 2005
9,030
214
Sorry notoriusjt2,

Mia culpa.

I think I deleted the thread since it appeared to be an accidental double post.

Can you repost the question?

hgmjr

Feb 4, 2010
209
0
no worries

format: plain


text:
# What does Pin type I mean on the ESP32 datasheet

On page 3 of the esp32-wroom datasheet (PDF) it shows a series of pins with type "I".

What does pin type I mean? Are these different to the pin type I/O pins? If so, how?

• It's Input. – Eugene Sh. Feb 27 '18 at 22:15
• @RenegadeAndy, what does I/O mean? – jsotola Feb 27 '18 at 22:16
• @jsotola Hm... One/Zero?? :) – Eugene Sh. Feb 27 '18 at 22:18
• @EugeneSh. So, undefined or infinity (if you are taking the limit)? – Samuel Feb 27 '18 at 22:23

Some pins can be input and output: I/O (or sometimes General Purpose I/O: GPIO).

Some pins can be outputs only: O.

Some pins can be inputs only: I.

Some pins can be power: P (note that some datasheets will differentiate between input power and ground).

And some pins can be not connected: NC.

• There is a P type as well. For Power.. – Eugene Sh. Feb 27 '18 at 22:23
• @EugeneSh. I didn't actually look at the datasheet... I know, for shame. – Samuel Feb 27 '18 at 22:26
• Sorry Samuel but I/O is a pin direction type, whereas GPIO is a pin label prefix used by microcontrollers. So they've distinctly different meanings and are not alternative names for the same thing. GPIO1 is the name of pin of type I/O, same as D7 is a pin of type I/O. – TonyM Feb 27 '18 at 22:47
• @TonyM No need to be sorry, they're not as mutually exclusive as you seem to imply with "distinctly different meanings". Both are of the direction category which is input/output, so yes, in some cases they are in fact the same thing. All squares are rectangles, but not all rectangles are squares. All GPIOs are I/Os, but not all I/Os are GPIOs. – Samuel Feb 27 '18 at 22:52
• Oh, I wasn't implying a point with "distinctly different meanings", I was firmly stating it. GPIO is a pin function and label for microcontrollers but it's not a direction. The General Purpose part describes a pin that a microcontroller's CPU can read or write. Logic chips, memories, FPGAs etc don't have GPIO pins, they have I/O pins. Important not to confuse the two, one's a noun and one's an adjective. – TonyM Feb 27 '18 at 23:17


text:
[–][S] 0 points1 point  (0 children)

Thank you, definitely worth checking out

[–][S] 0 points1 point  (0 children)

Two nights ago there was a Blastoise in front of ISA in the park, though I doubt it spawns there regularly as it is the first I have seen in a week

[–][S] 0 points1 point  (0 children)

No foolin? I could use some Charmanders!

[–] 13 points14 points  (0 children)

not Comcast

Pretty much enough for any Floridian to scream "SOLD!"

[–] 1 point2 points  (0 children)

I don't know what I expected, it is Ubisoft after all...

[–] 1 point2 points  (0 children)

I'm not sure but I feel your GPU might have a bit of trouble pushing reasonable frames at 1440p.

If nothing else I currently have an ASUS VG248QE which is 144Hz 1080p and I absolutely love it. I am even strongly considering grabbing a second one now.

Eventually I will get a 1440p 144Hz (eyeing the pg278q and another monitor I can't remember the name of at the moment) but trust me the 144Hz is hugely noticeable almost all the time and my favorite thing ever. Merely watching the cursor move on the 144Hz screen makes me just start smiling.

[–][S] 0 points1 point  (0 children)

Thank you very much for all of the information. Yeah Ideally I would want to wait to see how reliable it is and the like. It is a fairly large purchase so logistically I wouldn't feel comfortable purchasing it for AT LEAST another 3-4 weeks (or approximately 2 more paychecks for me) anyway.

I definitely appreciate any good source of information and reviews I can get. I don't want to buy it and have it burn out in a month or so you know.

Lately I have really been thinking I might buy a second 1080p monitor next week and give the monitor a few months to build up a reputation then I can decide it I want to replace my main display. By this time I would have a lot more in savings anyway. In the meantime if anyone else has any info on the 1440 monitor I would love to hear it

[–] 0 points1 point  (0 children)

That would be great and is also a major part of why I am holding out. To me it seems reasonable that a monitor, especially one plagued with so many problems, cost closer to $400-$450 but considering the others currently available the price isn't too bad, especially if it works well

[–] 0 points1 point  (0 children)

Unfortunately, not on the mobile app which I use pretty much exclusively. Weirder still, on the mobile app at least, the reviews don't even change.

Both get 4.5 stars at 61 reviews which again leads me to believe they have shared reviews (aside from reviews on one with pictures of the other monitor which fell into the obvious category before, they definitely do share them though)

As for the price, it is a HUGE plus. So many of the other options either cost $200-$400 more OR $100 less with almost 0 chance of it working. At$500 I am very strongly considering biting the bullet and ordering one sometime in the next two weeks.

The terrible RMA seems to be ASUS rather than ACER so at the very least it shouldn't be a terrible headache to get a replacement

[–] 0 points1 point  (0 children)

I was actually looking at getting one of those myself but don't know anyone who has one. At \$500 I just want to be sure I will not have to return 20 of them to get one that works proper.

Unfortunately Amazon ever so brilliantly has it share all of its reviews with the XG270HU so it is impossible to tell which feedback is actually for the XF while a LOT is clearly for the XG.

[–][S] 0 points1 point  (0 children)

Thank you for the info. I'm surprised you get such great performance still despite the increase in pixels while using a 770. This is definitely one I will consider at the very least. I'm very much hoping to find one with a bit of a lower price tag if possible but if not I could try for this one in two months or so (so I have enough money saved that it doesn't instantly make me broke). It was on my amazon wishlist as a possible one just that price is tough to live with

[–] 0 points1 point  (0 children)

Thank you very much mate, I was getting worried I wouldn't be able to play any more for a while. I was able to actually start a mission so, good news so far.

Out of curiosity, can someone explain exactly what you are doing by adding the hex offsets? I'm a little confused, is it a ROP-style work around?

EDIT: Oh I forgot to mention, for those of you like me who put the game in a different drive (E.G. have a hard drive for games but SSD for OS), first type your drive letter followed by a colon then cd to it and do the other stuff. For example, my games are on the E: drive so I do "E:" -> "cd path\to\pso2-bin" -> SET ....

EDIT 2: OK so if you install the english patch even this method doesn't seem to help. Crash after roughly 2 minutes in the game. I've been up all night tinkering so I'm off to bed. Hopefully it's fixed soon :(

[–] 1 point2 points  (0 children)

Oh man I honestly thought it was just me, I have been losing my mind because I just got 90 day premium and kicked off right at an alert seemingly randomly. I thought it was only me. I went so far as to totally delete and redownload the game. Hopefully the fixes are put in place soon

[–] 1 point2 points  (0 children)

Damn, I always assumed things like this took forever. I guess now I know it doesn't but I always found it funnier thinking people spent days on something so ridiculous.

[–] 0 points1 point  (0 children)

I also quite like the look of this mouse, the only thing I don't like (and I could he wrong) is that it lacks the 12 side buttons. I have two mice with them and they have become pretty much completely integrated into my daily use.

Not sure where all the hate is coming from.

[–] 5 points6 points  (0 children)

Lol mate, when you are used to barely getting 15, 30 IS good. I used to play ESO and fluctuate between 8 and 18 fps (most people would consider this entirely unplayable) and suffered through it then some optimizations brought me to a regular 36-38ish range and it was like a whole new world.

Now with the gtx 980 i have pumping out 80+ frames where not framelocked, it is fantastic

[–] 0 points1 point  (0 children)

Aw damn, doesn't show on the app I use. Post said 24 hours later and it had only been 22. Oh well maybe next time lol

[–] 1 point2 points  (0 children)

My brother recently got a 144Hz 1080p monitor and has been pushing me to get one. Now that I'm finally employed I am very strongly considering it because it looks so deliciously epic. I kind of goofed and didn't factor a monitor into my build (yes, stupid I know) and have been stuck playing on a 60Hz 32 inch lcd tv with 1600x900 resolution or, more frequently, a 60 Hz 15-ish inch monitor with a 1440x900 resolution.

Even when my brother just had a better rig than me and could run things at 80 fps when I was stuck getting barely 20 on my POS 650M, the difference was so vividly clear. The image on his end seemed so fluid whereas I was used to this gritty, choppy nonsense

[–] 0 points1 point  (0 children)

Nobody told me they have developed cases with antigrav technology, I want one!

[–] 0 points1 point  (0 children)

It seems last minute (unless this is over) but screw it

Entering

[–] -1 points0 points  (0 children)

10/10! That is actually some pretty impressive talent there. Much longer than I thought it would be starting out and it all works well too.

Well gents, it isn't terribly long now. T-minus a couple of hours until many of us make poor decisions with our money. Try not to go too broke out there, good luck everyone. Hopefully we can all get what we want.

[–] 3 points4 points  (0 children)

Plays your favorite tunes directly into your rectum to provide a groundbreaking feeling of syncing up to the mood of your music. Experience this highly sleek and innovative technology today!

Lightsabers represent by in pcmasterrace

[–] 0 points1 point  (0 children)

So a little bit random but anyone else on mobile see the Jimmy Johns logo on the AMD card at first glance? I can't help but see it every time I look at this pic