text:
Faced with continuously increasing scale of data, original back-propagation neural network based machine learning algorithm presents two non-trivial challenges: huge amount of data makes it difficult to maintain both efficiency and accuracy; redundant data aggravates the system workload. This project is mainly focused on the solution to the issues above, combining deep learning algorithm with cloud computing platform to deal with large-scale data. A MapReduce-based handwriting character recognizer will be designed in this project to verify the efficiency improvement this mechanism will achieve on training and practical large-scale data. Careful discussion and experiment will be developed to illustrate how deep learning algorithm works to train handwritten digits data, how MapReduce is implemented on deep learning neural network, and why this combination accelerates computation. Besides performance, the scalability and robustness will be mentioned in this report as well. Our system comes with two demonstration software that visually illustrates our handwritten digit recognition/encoding application.
We introduce a new structure for memory neural networks, called feedforward sequential memory networks (FSMN), which can learn long-term dependency without using recurrent feedback. The proposed FSMN is a standard feedforward neural networks equipped with learnable sequential memory blocks in the hidden layers. In this work, we have applied FSMN to several language modeling (LM) tasks. Experimental results have shown that the memory blocks in FSMN can learn effective representations of long history. Experiments have shown that FSMN based language models can significantly outperform not only feedforward neural network (FNN) based LMs but also the popular recurrent neural network (RNN) LMs.
We compare and contrast two approaches to validating a trained classifier while using all in-sample data for training. One is simultaneous validation over an organized set of hypotheses (SVOOSH), the well-known method that began with VC theory. The other is withhold and gap (WAG). WAG withholds a validation set, trains a holdout classifier on the remaining data, uses the validation data to validate that classifier, then adds the rate of disagreement between the holdout classifier and one trained using all in-sample data, which is an upper bound on the difference in error rates. We show that complex hypothesis classes and limited training data can make WAG a favorable alternative.
An experimental approach to studying the properties of word embeddings is proposed. Controlled experiments, achieved through modifications of the training corpus, permit the demonstration of direct relations between word properties and word vector direction and length. The approach is demonstrated using the word2vec CBOW model with experiments that independently vary word frequency and word co-occurrence noise. The experiments reveal that word vector length depends more or less linearly on both word frequency and the level of noise in the co-occurrence distribution of the word. The coefficients of linearity depend upon the word. The special point in feature space, defined by the (artificial) word with pure noise in its co-occurrence distribution, is found to be small but non-zero.
Boosting is a generic learning method for classification and regression. Yet, as the number of base hypotheses becomes larger, boosting can lead to a deterioration of test performance. Overfitting is an important and ubiquitous phenomenon, especially in regression settings. To avoid overfitting, we consider using $l_1$ regularization. We propose a novel Frank-Wolfe type boosting algorithm (FWBoost) applied to general loss functions. By using exponential loss, the FWBoost algorithm can be rewritten as a variant of AdaBoost for binary classification. FWBoost algorithms have exactly the same form as existing boosting methods, in terms of making calls to a base learning algorithm with different weights update. This direct connection between boosting and Frank-Wolfe yields a new algorithm that is as practical as existing boosting methods but with new guarantees and rates of convergence. Experimental results show that the test performance of FWBoost is not degraded with larger rounds in boosting, which is consistent with the theoretical analysis.
A thesis submitted for the degree of Doctor of Philosophy of The Australian National University. In this work we introduce several new optimisation methods for problems in machine learning. Our algorithms broadly fall into two categories: optimisation of finite sums and of graph structured objectives. The finite sum problem is simply the minimisation of objective functions that are naturally expressed as a summation over a large number of terms, where each term has a similar or identical weight. Such objectives most often appear in machine learning in the empirical risk minimisation framework in the non-online learning setting. The second category, that of graph structured objectives, consists of objectives that result from applying maximum likelihood to Markov random field models. Unlike the finite sum case, all the non-linearity is contained within a partition function term, which does not readily decompose into a summation. For the finite sum problem, we introduce the Finito and SAGA algorithms, as well as variants of each. For graph-structured problems, we take three complementary approaches. We look at learning the parameters for a fixed structure, learning the structure independently, and learning both simultaneously. Specifically, for the combined approach, we introduce a new method for encouraging graph structures with the ‘scale-free’ property. For the structure learning problem, we establish SHORTCUT, a O(n^{2.5}) expected time approximate structure learning method for Gaussian graphical models. For problems where the structure is known but the parameters unknown, we introduce an approximate maximum likelihood learning algorithm that is capable of learning a useful subclass of Gaussian graphical models.


text:
Resource type: this resource contains a tutorial or tutorial notes.
 Completion status: this resource is considered to be complete.

This tutorial teaches use of analysis of covariance (ANCOVA) techniques, with practical exercises based on using SPSS.

## Overview

• An ANCOVA evaluates whether population means on the DV, adjusted for differences on the covariate(s) (or 'nuisance variables'), differ across the levels of the IVs. Thus, the question being tested is whether the adjusted group means vary significantly from each other.
• ANCOVA is exactly like ANOVA, except the effects of a third variable are statistically “controlled out” (similar to use of hierarchical multiple linear regression).
• Any number of IVs and CVs can be used to create one-way, two-way, and multivariate ANCOVA designs.

## Covariates

• Typically included in an experimental design to remove extraneous influences from the DV, thus decreasing the within-group variance. Particularly with small sample sizes, well-chosen and reliably measured CVs can markedly improve the power of inferential statistical tests.
• Including covariates is appropriate in order to:
1. Eliminate some systematic variance outside the control of the researcher that can bias the results.
2. Account for differences in response due to unique characteristics of the respondents.
This is usually achieved in experimental designs by random assignment to groups, however, in quasi-experimental designs problems related to non-random assignment can be minimised by statistically controlling for the effects of covariates.
• Try to minimise the number of CVs; too many covariates will reduce the statistical efficiency of the analysis - rule of thumb is that the number of CVs < (.10 x sample size) - (number of groups - 1).

### Example

If you are interested in testing the effect of computer experience on the attitude towards use of internet shopping, and you suspect that those with more positive attitudes toward shopping in general are more likely to have positive attitudes towards internet shopping, you may include attitude toward shopping as a covariate so as to remove its influence from the attitude towards internet shopping measure.

## Assumptions

Assumptions to be met are those for ANOVA, plus:

1. Covariates must be linearly related to the DV. The stronger the correlation, the more useful the CV will be. If there is no correlation, then the inclusion of the CV will slightly weaken the power of the test by needlessly consuming a degree of freedom.
2. Covariates must have a homogeneity of regression (or homogeneity of slopes) effect (equal effects on the DV across the IV groups) - if there is a significant interaction between the covariate and the factor – you cannot use this procedure.
3. The covariate should be unrelated to the IV.
4. Covariates should not be overly correlated with one another.

The first two criteria can be checked via a scatterplot of the DV and the CV, with the IV as control variable (to check for equal slopes)

The third criteria depends on experimental design, e.g., if the CV is measured prior to the IV, then it cannot be affected by the IV.

The fourth criteria can be checked via correlations and scatterplots between the CVs.

The main ANOVA assumptions are that:

1. Each of the observations are independent.
2. The DV (and the CV) must be interval level of measurement.
3. The underlying populations (of adjusted scores) must be normally distributed.
4. Each of the underlying populations (of adjusted scores) must have the same variance.

The first two assumptions are a function of experimental design. The third and fourth assumptions are more difficult to test because we do not have the 'adjusted scores', so we cannot compare the variances of the adjusted scores across the IV groups, but SPSS provides a Levene's test for the homogeneity of variance for the adjusted scores. Fortunately, provided the samples are sufficiently large, the test is robust to violations of the normality assumption.

## Example write-up

A one-way analysis of covariance (ANCOVA) was conducted. The independent variable, vitamin C, involved three levels: placebo, low dose, and high dose. The dependent variable was the number of days with cold symptoms during treatment and the covariate was the number of days with cold symptoms before treatment. The assumptions for ANCOVA were met. In particular, the homogeneity of the regression effect was evident for the covariate, and the covariate was linearly related to the dependent measure.

The ANCOVA was significant, F (2,26) = 6.45, p = .005. The strength of the relationship between vitamin C treatment and the dependent variable was very strong, as assessed by ${\displaystyle \eta }$${\displaystyle _{p}^{2}}$, with the vitamin C factor accounting for 33 percent of the variance in dependent measure holding constant the number of days with pretreatment cold symptoms. The mean number of days with cold symptoms adjusted for initial differences were ordered as expected across the three vitamin C groups. The placebo group had the largest adjusted mean (M = 12.01), the low dose vitamin C group had a smaller adjusted mean (M = 7.71) and the high dose vitamin C group had the smallest adjusted mean (M = 6.67). Follow-up tests were conducted to evaluate pairwise differences among the adjusted means. The Holm’s sequential Bonferroni procedure was used to control for Type I error across the three pairwise comparisons. There were significant differences in the adjusted means between both groups that received vitamin C and the placebo, but no significant difference between the two vitamin C groups. [Standardised mean effect sizes to be added] e.g., effect size with covariate calculator

## Descriptives

In presenting ANCOVA results, provide a table of means for each of the groups. If the same scale is used to measure the DV and the CV, then the unadjusted group means and SD (from Descriptives) can be presented. If a different scale is used for the DV and CV, then provide both the unadjusted mean (and SD) and the adjusted mean (and SE). The adjusted mean (controlling for the CV) is provided in the Estimated marginal means table.

## Exercises

### Teaching method

In a hypothetical educational psychology experiment, participants were randomly divided into two groups. One group was taught conventionally, and the other were taught using an innovative method. Prior to allocation to the groups, learning motivation was assessed. Improvements in academic achievement were measured as the DV.

• Motiv.sav (p. 129; Section 5.2)
• Achievement (achieve) (DV)
• Teaching Method (teach) (IV - 2 levels - "conservative" and "innovative")
• Motivation (motiv) (CV)

### Positive effect

Conduct an ANCOVA to test for differences in positive effect between males and females, adjusting for differences in age.

### Vitamin C

Do low and high doses of Vitamin C reduce incidences of days suffered with a cold? Conduct an ANOVA using:

• T ANCOVA.sav
• Group (IV; Vitamin C: placebo, low, high)
• Days (DV; days with a cold)
• Pre-days (CV; previous days with a cold)

### Therapy and depression

What is the effect of three different therapy types on depression, taking into account pre-existing depression levels? Use:

• Ex ANCOVA.sav
• Group (IV; Counseling and journal therapy; Journal therapy only; Counseling only)
• Depression prior
• Depression after

format: plain


text:
# Real Polynomial Function is Continuous

## Theorem

A (real) polynomial function is continuous at every point.

Thus a (real) polynomial function is continuous on every interval of $\R$.

## Proof 1

Let $f_n$ be an arbitrary real polynomial function of degree $n$.

The proof proceeds by induction.

For all $n \in \Z_{\ge 0}$, let $\map P n$ be the proposition:

$f_n$ is continuous on $\R$.

$\map P 0$ is the case $f_0$, where $f_0$ is of zero degree.

Such a real polynomial function is a constant function.

It follows from Constant Real Function is Continuous that $f_0$ is continuous on $\R$.

Thus $\map P 0$ is seen to hold.

### Basis for the Induction

$\map P 1$ is the case:

$f_1$ is continuous on $\R$.

Such a real polynomial function is a linear function.

Thus $\map P 1$ is seen to hold.

This is the basis for the induction.

### Induction Hypothesis

Now it needs to be shown that if $\map P k$ is true, where $k \ge 1$, then it logically follows that $\map P {k + 1}$ is true.

So this is the induction hypothesis:

An arbitrary real polynomial function of degree $k$ is continuous on $\R$

from which it is to be shown that:

An arbitrary real polynomial function of degree $k + 1$ is continuous on $\R$.

### Induction Step

This is the induction step:

$\forall x \in \R: \map {f_{k + 1} } x = a_{k + 1} x^{k + 1} + \map {f_k} x$

for some real polynomial function of degree $k$, where $a_{k + 1} \in \R$

From the induction hypothesis $f_k$ is continuous on $\R$.

From Real Power Function for Positive Integer Power is Continuous, the real function $x \mapsto x^{k + 1}$ is continuous.

Hence from the Combined Sum Rule for Continuous Real Functions:

$f_{k + 1}$ is continuous.

So $\map P k \implies \map P {k + 1}$ and the result follows by the Principle of Mathematical Induction.

Therefore:

$\forall n \in \Z_{\ge 0}: f_n$ is continuous on $\R$.

$\blacksquare$

## Proof 2

From Linear Function is Continuous, setting $\alpha = 1$ and $\beta = 0$, we have that:

$\ds \lim_{x \mathop \to c} x = c$

Repeated application of the Product Rule for Limits of Real Functions shows us that:

$\ds \forall k \in \N: \lim_{x \mathop \to c} x^k = c^k$

Now let $\map P x = a_n x^N + a_{n - 1} x^{n - 1} + \cdots + a_1 x + a_0$.

By repeated application of the Combined Sum Rule for Limits of Real Functions, we find that:

$\ds \lim_{x \mathop \to c} \map P x = \map P c$

So whatever value we choose for $c$, we have that $\map P x$ is continuous at $c$.

From the definition of continuity on an interval, the second assertion follows.

$\blacksquare$


text:
On Local Regret
Online learning typically aims to perform nearly as well as the best hypothesis in hindsight. For some hypothesis classes, though, even finding the best hypothesis offline is challenging. In such offline cases, local search techniques are often employed and only local optimality guaranteed. For online decision-making with such hypothesis classes, we introduce local regret, a generalization of regret that aims to perform nearly as well as only nearby hypotheses. We then present a general algorithm that can minimize local regret for arbitrary locality graphs. We also show that certain forms of structure in the graph can be exploited to drastically simplify learning. These algorithms are then demonstrated on a diverse set of online problems (some previously unexplored): online disjunct learning, online Max-SAT, and online decision tree learning.


text:
# Finiteness properties for graph of groups decompositions

My curiosity was raised by the following question and the huge variety of comments and suggestions it attracted. I wondered if a converse statement might be equally interesting.

Let $G$ be a finitely presented group which admits a non-trivial graph of groups decomposition. Does it follow that each vertex stabiliser is finitely presented? How about if every edge stabiliser is finite?

Dunwoody comments that if $G$ is finitely generated and each edge stabiliser is finite then each vertex stabiliser is finitely generated.

• A silly example: Take $F= \langle a_1,a_2,\ldots \mid \ \rangle$ a free group of infinite rank, and $f : a_i \mapsto a_{i+1}$ be an isomorphism between the subgroups $\langle a_1,a_2, \ldots \rangle$ and $\langle a_2,a_3, \ldots \rangle$. Then, the associated HNN extension is just $$\langle t, a_1,a_2, \ldots, \mid ta_1t^{-1}=a_2, ta_2t^{-1} =a_3,\ldots \rangle \simeq \langle t \mid \ \rangle \simeq \mathbb{Z}.$$ – Seirios Feb 27 '15 at 17:40
• In my example I assumed the vertex stabilizers to be finitely generated, indeed otherwise there are even easier examples. – YCor Feb 27 '15 at 18:55

Just notice that there exist infinitely presented groups $H$ with an automorphism such that the corresponding semidirect product $G=H\rtimes\mathbf{Z}$ is finitely presented. This is a non-trivial graph of groups decomposition, namely an HNN-decomposition where the vertex group and edge group coincide; in case you find this too degenerate, just add some further flourish to the graph of groups, so that $H$ still appears as a vertex group and the resulting group is still finitely presented. One example is when $G=BS(1,2)^2=\mathbf{Z}[1/2]^2\rtimes\mathbf{Z}^2$ where the generators of $\mathbf{Z}$ act by the diagonal matrices $(2,1)$ and $(1,2)$, and $H=\mathbf{Z}[1/2]^2\rtimes\mathbf{Z}$ where $\mathbf{Z}$ acts by the diagonal matrix $(2,1/2)$.
On the other hand, for the second question "How about if every edge stabiliser is finite?", the answer is yes. Indeed, write $H$ as a limit of truncated presentations $H_n$ (this limit here means: inductive limit of surjections between f.g. groups). For $n$ large enough (say $n\ge n_0$), all finite subgroups of $G$ occurring in the graph of groups decomposition of $G$ are well-defined, and this allows to write $G$ as (non-attained) limit of corresponding graph of groups $G_n$ for $n\ge 0$, where in the graph of groups, $H$ was replaced with $H_n$). The argument still works if the edge groups are finitely presented instead of finite.


text:
# Solution

In the arts, the golden rectangle, often considered to be the rectangle with the most aesthetically pleasing proportions, has the property that if you remove a square whose edge length matches that of the smallest side of the rectangle (shown in blue below), you are left with a smaller rectangle (shown in red) with the same proportions as the original one. That is to say $(a+b)/a = a/b$.

Find the value of $a/b$, dubbed the golden ratio and its reciprocal -- how are they related?

We know that

$$\frac{a+b}{a} = \frac{a}{b}$$

If we wish to solve for $a/b$, it would be helpful if this was the only unknown in the equation. That is to say, if $x=a/b$, can we rewrite our equation so that it is in terms of only $x$?

We can turn the $a$ in the denominator into an $a/b$ by dividing it by $b$. So that we don't alter the value of the expression on the left, let us do this to the numerator as well, yielding

$$\frac{a/b + 1}{a/b} = \frac{a}{b}$$

Now, we can replace each $a/b$ with $x$ as suggested above, and then simply solve for $x$

$$\frac{x+1}{x} = x$$

Clearing the fractions by multiplying by $x$, we have

$$x+1 = x^2$$

This of course is quadratic, so its solution is routine.

$$x^2-x-1=0$$

which yields

$$x = \frac{1 \pm \sqrt{5}}{2}$$

Knowing that $x = a/b$ is a ratio of side lengths, it must be true that $x>0$, so we can eliminate the negative solution found. Thus, the remaining solution must be the value of the golden ratio, $\varphi$, that we seek:

$$\varphi = \frac{1 +\sqrt{5}}{2}$$


text:
# How do you factor w^2+4w+3?

Then teach the underlying concepts
Don't copy without citing sources
preview
?

#### Explanation

Explain in detail...

#### Explanation:

I want someone to double check my answer

1
Mar 7, 2018

$\left(w + 3\right) \left(w + 1\right)$

#### Explanation:

In this case, since $a = 1$ (in $a {x}^{2} + b x + c$), simply look for two numbers whose sum is $4$ and whose product is $3$. You can easily see that the two numbers are $1$ and $3$.

Then, just write it down: $\left(w + 3\right) \left(w + 1\right)$

Check: try expanding the brackets using FOIL.

• 23 minutes ago
• 24 minutes ago
• 26 minutes ago
• 27 minutes ago
• A minute ago
• 5 minutes ago
• 17 minutes ago
• 20 minutes ago
• 21 minutes ago
• 21 minutes ago
• 23 minutes ago
• 24 minutes ago
• 26 minutes ago
• 27 minutes ago


text:
Graphs and Groups, Geometries and GAP (G2G2) Summer School - External Satellite Conference of 8ECM

27 June 2021 to 3 July 2021
Rogla, Slovenia
UTC timezone

Type IV-II codes over $\mathbb{Z}_4$ constructed from generalized bent functions

Not scheduled
20m
Rogla, Slovenia

Rogla, Slovenia

Hotel Planja, Rogla 1, 3214 Zreče, Slovenia
Oral presentation

Speaker

Dr Sara Ban (Department of Mathematics, University of Rijeka)

Description

A Type IV-II $\mathbb{Z}_4$-code is a self-dual code over $\mathbb{Z}_4$ with the property that all Euclidean weights are divisible by eight and all codewords have even Hamming weight.
The subject of this talk is a construction of Type IV-II codes over $\mathbb{Z}_4$ from generalized bent functions.

We use generalized bent functions for a construction of self-orthogonal codes over $\mathbb{Z}_4$ of length $2^m$, for $m$ odd, $m\geq3$, and prove that for $m\geq5$ those codes can be extended to Type IV-II $\mathbb{Z}_4$-codes. From that family of Type IV-II $\mathbb{Z}_4$-codes, we construct a family of self-dual Type II binary codes by using the Gray map.

We also consider the weight distributions of the obtained codes and the structure of the supports of the minimum weight codewords, which we use for a construction of 1-designs.
Some of the constructed 1-designs are affine resolvable 1-designs.
For the constructed 1-designs, we examine the properties of the corresponding block intersection graphs and obtain strongly regular graphs in two cases.

Primary authors

Dr Sara Ban (Department of Mathematics, University of Rijeka) Prof. Sanja Rukavina (Department of Mathematics, University of Rijeka)


text:
# Counting the number of $(d_v,d_c)$ regular bipartite graphs

I am trying to count the number of $(d_v,d_c)$ regular bipartite graphs. To be specific, let $n,m,d_v,d_c$ be positive integers such that $$n\times d_v=m\times d_c.$$ Then, what is the number of bipartite graphs $\mathcal{G}=(L\cup R, E)$, where $L$ is the set of left vertices, $R$ is the set of right vertices with the property that each left vertex has $d_v$ edges incident on it, and each right vertex has $d_c$ edges incident on it (allowing parallel edges, i.e., there can be more than one edge for a given pair of vertices)? Can one find a general expression for this?

This problem can be reformulated as a ball and bin problem in the following way: We have a total of $nd_v$ balls, of $n$ colors, with $d_v$ balls of each color. What is the number of ways of putting these balls in $m$ nonidentical bins so that each bin has exactly $d_c$ balls? Let me denote this number $\psi(n,d_v,d_c)$ A quick thought about this gives me the following bounds: $$\frac{(nd_v)!}{\left(d_v! \right)^n(d_c!)^m}\leq\psi(n,d_v,d_c)\leq \frac{(nd_v)!}{\left(d_v! \right)^n}.$$ Can one do better?

I think there must be some work that has addressed this problem but I have not been able to find anything. Any help would be appreciated.

There is no useful exact formula except for tiny $d_v$ or $d_c$. The best asymptotic results appear in this paper of Canfield and McKay for dense matrices and this paper of Greenhill and McKay for sparse matrices. There is an unsolved gap between the two ranges. The first paper has a conjecture that probably holds for all values of the parameters.


text:
# conformal, Weyl transformations, apparent discrepancies and confusions

Because of the apparent discrepancy of how some CFT and GR books define conformal transformation unlike in string theory area, I wanted to get rid of all the confusion from McGreevy's lecture notes:

http://ocw.mit.edu/courses/physics/8-821-string-theory-fall-2008/lecture-notes/lecture10.pdf

I have understood and happy with the definition of conformal group and conformal transformation now, but I have the following two questions:

1. In page 2, where he defined conformal invariance, did he mean invariance under the diffeomorphism part only (which gives the conformal group)? The reason I think so is that he changed the metric which according to his definition of conformal transformation don't change. So, the reason I am asking is that I thought conformal invariance will be a suitable name for a theory, invariant under conformal transformation.

2. Just to make sure: the transformation in page 2: $\delta g_{\mu\nu}=2\lambda g_{\mu\nu}$ and $\delta g_{\mu\nu}=\delta\Omega(x)\delta g_{\mu\nu}$, these are results of diffeomorphism ..right? and not Weyl in any way. In other words, what are these transformations?

All my questions are for general dimension and general space time (classically of course).

-

It's somewhat unclear how you "understood" and are "happy" about the definition of the conformal transformations because your questions, while referring to page 2 and other things, are nothing else than misunderstandings about the definition of a conformal transformation which is explained on page 1, not 2.

John wrote his equation 1 which states that conformal transformations "are" diffeomorphisms that only change the metric by a local scalar coefficient - by a Weyl rescaling. However, the invariance of a theory under these diffeomorphisms is a trivial property. If one is allowed to change the terms coupled to the "metric", a diffeomorphism-transformed theory has a different action in general, and it is always possible to rewrite the original action in the conformally transformed coordinates.

But what's physically nontrivial is the condition that the action, in its original form, is actually invariant under the operations - that's what we mean by the theory's being conformally invariant. If a theory is conformally invariant, we don't allow any "change of the coefficients" in the integral of the Lagrangian density. This condition of conformal invariance, as he shows, is equivalent to the invariance of the theory under the "Weyl rescaling" only: we just completely eliminate the diffeomorphisms from the picture.

So:

1. Again, the invariance under the combined "diffeomorphism" and "Weyl rescaling" (the latter changes the form of the action) is a tautology. Obviously, by conformal invariance, we don't mean a tautology, so by conformal invariance, we mean the invariance of the action under the diffeomorphism separately, without changing the form of the metric in the action. Because the invariance under the "combo" is tautological, conformal invariance is equivalent to the invariance under the "Weyl rescaling part" of the transformation only.

2. No, on page 2, the transformations are exactly what the equations say: point-dependent transformations of the metric tensor itself i.e. a Weyl rescaling. There is no diffeomorphism at this stage. The invariance under those Weyl transformations is equivalent to the invariance under some diffeomorphisms (with the modification of the metric erased), as explained in the previous point.

The formalism may depend on classical physics but your comment that it is "classically only of course" is incorrect, too. All those facts about conformal transformations are completely valid quantum mechanically as well - and indeed, this background is primarily mean to understand some quantum theories.

-


text:
## Friday, April 29, 2016

### Excel: Simplex vs Evolutionary Engine (Simplex wins)

In this post a simple MIP problem is posted. Basically the problem can be stated as:

 \boxed{\begin{align} \max & \sum_i Score_i \, x_i \\ & \sum_i x_i = 6 \\ & \sum_i Salary_i \, x_i \le 50000 \\ & x_i \in \{0,1\} \end{align}}

We have a database of 153 players, and we want to compose the best team of 6 players with a total salary cap of \$50k. This can actually be done in Excel quite easily. With some random data I get the following results using the Simplex LP based MIP solver:

The name “Simplex LP” is somewhat misleading, as we solve here a MIP.

Some comments in the post are really surprising to me:

• the number of combinations of 6 golfers from a pool of 153 is 16,133,132,940 Note that yes, that is over 16 billion combinations. Even if Solver can do it, it will probably crash Excel”. A MIP solver (or the Evolutionary Solver) will not evaluate all possible solutions! (That would be impossible in almost all cases). There is no reason to believe Excel will crash on this problem (quite the opposite: it solves quite quickly). I see this argument more often; apparently there are lots of people around who think complete enumeration is the best we can do.
• Excel mentions “use Evolutionary engine for non-smooth problems” so one answer suggested to use the Evolutionary solver. Actually I understand why someone may think this is a wise decision: the hint can be interpreted as advice to use the Evolutionary algorithm for any discrete problem. Looking at the results below this is not such a good idea: the Evolutionary engine takes much more time and delivers a significantly sub-optimal solution. A summary comparison of the results is:

• The built-in version of Solver reports, ‘This problem is too large for Solver to handle.’ [..] an alternative is the free OpenSolver add-in (see opensolver.org)“ I have no idea why this user sees this message. I had no size problems using the sizes as indicated by the poster. This problem has 153 binary variables and I believe the limit is 200 variables.

Below is the result from a run with the same data using the Evolutionary solver:

#### Conclusion

My advice: always try a MIP model first. The model building process forces you to understand the problem better and a MIP solver can give proven optimal solutions (or a guaranteed quality when we allow a gap between best possible and best found solutions). If the performance is not satisfactory then consider using a meta-heuristic. (The MIP model an also be used then to debug the implement the meta-heuristic and compare the quality of the objective function values).


text:
• FEATURED

View more

View more

### Image of the Day Submit

IOTD | Top Screenshots

### The latest, straight to your Inbox.

Subscribe to GameDev.net's newsletters to receive the latest updates and exclusive content.

## Inconsistent shadows from a directional light

Old topic!

Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

3 replies to this topic

### #1sotteson  Members

Posted 22 August 2013 - 11:56 PM

I'm writing a D3D game where the player's ship flies along and casts a shadow nearly directly underneath to give a clue to the player on the height of the ship. I'm attempting to make this a directional light by using an orthogonal projection for the projection matrix of the single light. As the ship moves forward, I update the location of the light so that the shadow remains the same relative to the ship's position. The problem is because I move the light, shadows for other objects move relative to the light, instead of also staying underneath their objects.

(I'm purposely darkening the shadows so that you can see them better.)

The blue ship is moving forward and the gray ship is stationary. Note where the shadow is for the gray ship.

The blue ship has moved forward. I've moved up the light so that the shadow stays under the blue ship, but I need the shadow under the other ship to stay under itself too. Instead, its shadow moved up since I moved up the light to chase the blue ship.

I feel like this should be easy to fix, but so far I can't figure it out. Thanks for your help.

Steve

### #2Norman Barrows  Members

Posted 23 August 2013 - 05:51 AM

looks like whatever engine you're using is casting shadows for you (nice!).  but your light is way too close for it to seem like the sun. you can try moving the light farther away (like 93 million miles farther away) to get the correct effect. may not work though. it that case you fake it and draw the shadows yourself.   changing the light type may also work (directional vs point, etc).

Norm Barrows

Rockland Software Productions

"Building PC games since 1989"

rocklandsoftware.net

PLAY CAVEMAN NOW!

http://rocklandsoftware.net/beta.php

### #3sotteson  Members

Posted 23 August 2013 - 10:33 AM

I'm using raw DirectX 11 with my own shaders for this game. I tried moving the light to be 93 millions miles up, but then I still need to give it a z component that matches up with the player's ship. This continues to give the effect of the shadow directly under the player's ship, but shadows for other objects not staying underneath themselves. I may have to find a way to fake it unless anyone has any better ideas.

### #4sotteson  Members

Posted 07 September 2013 - 08:47 PM

I ended up switching to volumetric stencil shadows to get a shadow directly under everything that has a shadow. Right now the perf is acceptable, and hopefully it stays that way as I add more objects that cast shadows in the game.

Old topic!

Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.


text:
• Advertisement

#### Archived

This topic is now archived and is closed to further replies.

# Simple, Simple Questions

This topic is 5991 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

## Recommended Posts

After finishing "An Absolute Beginner''s Guide to C" I picked up "The C Programming Language". I have gone through the first few chapters and I understand most of it, but two things are confusing me, and if you could answer them it would be much appreciated. First off, can someone explain EOF a little better. All I caught was that it was an integer defined in ?(stdio.h)?, I forget, but I don''t see what the purpose of it is, and whether it can make the first few programs not an endless loop. And I have no idea what bitwise operators are doing. If anyone could explain the bits thing or direct me somewhere where I could it would be helpful. Thanks.

#### Share this post

##### Share on other sites
Advertisement
EOF = End of File. For example:
  while ( x != EOF ){ file.get(x); // read in a new character }

It's just a way of checking to see if you've hit the end of the file.

Bitwise operators are operators the work on individual bits. For example the number 7 in binary is 0000 0111. If we do a bitwise AND with 6(0000 0110) we do an AND on all the individual bits.

0000 0111
0000 0110
-----------
0000 0110

So 7 & 6 = 6

The are most often used as flags. For example:
  #define FLAG_1 1#define FLAG_2 2#define FLAG_3 4#define FLAG_4 8//Set a test to both FLAG_1 and FLAG_4int testFlag = FLAG_1 | FLAG_4;// 0000 0001 OR// 0001 0000//------------// 0001 0001// testFlag will be 0001 0001// Test to see if FLAG_4 has been setif ( testFlag & FLAG_ 4 ){ //0001 0001 AND// 0001 0000// ------------// 0001 0000 // Thus the test condition is true therefore FLAG_4 has been set}

Hope this helps a little. Try going to google and typing in "Bitwise operators in C++ tutorial" I'm sure that you will get lots of hits with better examples

-------------
Andrew

Edited by - acraig on August 25, 2001 2:18:51 PM

#### Share this post

##### Share on other sites

• Advertisement


text:
pchez305
7

# club A charge $11 for membership and$11 monthly. Club B charge $27 membership and$7 monthly. After how many many months will they charge the same? How much for each?

Let $m$ be the number of months. Since Club A charges $11 only one time and charges$11 per month, the total charge after $m$ months is $\11 + \11m$. Similarly, the total charge after $m$ months for Club B is $\27 + \7m$. Since we want to find after how many months the two clubs charge the same amount, we can set those two expressions equal to each other and solve: $\11 + \11m = \27 + \7m$ $\4m = \16$ $\bf m = 4$ So, Club A and Club B will charge the same amount after 4 months. To find how much they charge, just plug $m$ back into one of the original equations: $\11 + \11m = \11 + \11(4) = \bf \55$


text:
# 20140115 (Wednesday, 15 January 2014)¶

The author of sphinxcontrib-taglist helped me to understand how to use it.

His explanation:

You have:

.. tag:: blog docs

but it should be like:

.. tag:: [blog docs] Some text

or:

.. tag:: Some text
:tag: blog docs

The main idea was to ‘tag’ some piece of information, not the page as whole. So you need provide ‘Some text’ which you want to tag.

## What I really want¶

But then I realized that I want even more (or maybe less): I don’t want to specify a text for each occurence of a tag. I “just” want a directive that inserts a list of “pages referring to this”.

For example when I write in the present blog entry the present sentence which mentions Database Migration or Renamed “Lino Faggio” to “Lino Voga” (which for me is are well-known concepts, and I know that I have somewhere an article about it), then I want on both of these pages, after having explained that concept, in a section “Pages referring to this”, a reference to this blog entry. A simulation:

Data migration

Data migration is bla bla bla

Pages referring to this

docs/tickets/80


text:
# Math Help - limits

1. ## limits

May I in the following cases:

1. lim x=>a(2x2+3x+1)/x2=2
2. lim x=>a(-2x+(2x2+3x+1)/x)=3
a is infinity.

simply first divide by the denominator and then let x goes to infinity or, am I suppose first to take x out of the numerator then cancel the x(both at the numerator and denominator ) and just then let the x goes to infinity?


text:
# Calculating European call option, the Bjork way

We have a 3 period binomial tree with values:

                                              59.65 (C33 = 7.65)
56.24 (C22 = ?)
53.03                        53.03 (C32 = 1.03)
50                          50    (C21 = ?)
47.14                        47.14 (C31 = 0)
44.45 (C20 = ?)
41.91 (C30 = 0)

W want to calculate a European call option, no arbitrage, with properties K = 52, u = 1.0606, d = 1/u = 0.943, maturity in 9 weeks, r = 0.001 per week. The value for a call option is given by $$max[S_t -K, 0]$$. We can calculate $$C_2^{2}$$ given the risk neutral formula from the literature (Bjork 3ed, 2.1.4):

$$C_2^{2} = \frac{1}{1+R} (q*C_3^{3} + (1-q)*C_3^{2})$$, $$\frac{1}{1+R}$$, given by Bjork Proposition 2.6, but as we have multiple nodes I assume we need to discount it, which gives the formula $$e^{r-(T-t)}= e^{0.053348-(9/52)} = 1.009276$$

$$R = 1.001^{52} = 5.3348pct = 0.053348 ,$$

$$q = \frac{(1+R)-d}{u-d} = \frac{1.009276 - 0.943}{1.0606 - 0.943} = 0.5636$$, if we then plug the values into the formula:

$$C_2^{2} = \frac{1}{1+R} (q*C_3^{3} + (1-q)*C_3^{2}) = 1.009276*(0.5636*7.65 + 0.4363*1.03) = 4.8051$$,

My questions are:

a) is the value of $$C_2^{2}$$ correct?

b) is there a faster way to calculate the option value of the tree because this takes a lot of time (yes you can write a program but I am following the theory and I believe I have to learn it by hand as well).

• I think you need to fix your discounting in a few places. In general, multiplying by $1+r(T-t)$ is similar to multiplying by $e^{r(T-t)}$. You are mixing those up here. Sep 12 '20 at 22:50
• When I follow the method given by @RRL, it seems that the discount factor is $\frac{1}{(1+r)^9} = \frac{1}{(1+0.001)^9} = 0.9910$, can you point out where the mistakes are? Sep 13 '20 at 7:40
• It depends on if you do exponential or periodically-compounded discounting. I've seen trees built with both, but Bjork is using one so for replication you should use that. Sep 13 '20 at 7:52
• Are you referring to proposition 2.11? Sep 13 '20 at 7:58

Is there a faster way to calculate the option price?

With a recombining binomial tree, the terminal asset price has a binomial distribution -- as you might have expected. For a tree with $$n$$ steps, the probability of reaching price $$S_{n,k}$$ where $$k$$ is the number of up moves is

$$P_{n,k} = \frac{n!}{k!(n-k)!}q^k(1-q)^{n-k}$$

The option price is the discounted risk-neutral expectation of the payoff,

$$C = \frac{1}{(1+r_s)^n}\sum_{k=0}^n\frac{n!}{k!(n-k)!}q^k(1-q)^{n-k} \max(S_{n,k}-K,0),$$

where $$r_s$$ is the interest rate per period associated with a single step. Using this formula avoids working backwards and computing option values at intermediate steps.

In this case we have $$n= 3$$ and $$(1+r_s) = (1+0.001)^3$$ (since each step spans 3 weeks). Hence, since $$C_{31} = C_{30} = 0$$,

$$C = \frac{1}{(1+r)^9} (1 \cdot q^3 C_{33} + 3 \cdot q^2(1-q) C_{32})$$

(Note that the coefficient $$1$$ for the first term arises because there is one path through the tree reaching the node $$(3,3)$$ and the coefficient $$3$$ for the second term arises because there are three paths through the tree reaching the node $$(3,2)$$.)

• Many many thanks: if we apply your formula $C = \frac{1}{(1+r)^9} (1 \cdot q^3 C_{33} + 3 \cdot q^2(1-q) C_{32})$ = $\frac{1}{(1+0.001)^9} (1 \cdot 0.5636^3 (7.65) + 3 \cdot 0.5636^2(0.4363)(1.03))$ = $0.9910 \cdot ((0.1790 \cdot 7.65) + (3 \cdot (0.3176 \cdot 0.4363)1.03)) = 1.4918$ as the option price. Is it possible that you check the value of the price? Also because @kurtosis pointed out that my discounting is not correct Sep 13 '20 at 7:34
• You’re welcome. First, to get the risk neutral probability you have the correct formula $q = (1+R-d)/(u-d)$ but the $R$ here must be the interest rate for the single step period. It seems you are not given a continuously compounded rate but rather a weekly compounded rate $r = 0.001$. Since a single step is 3 weeks we have $R = (1+r)^3 -1$.
– RRL
Sep 13 '20 at 14:08
• Look at the Wikipedia entry for compound interest and in particular the example when a monthly rate is specified. There is a distinction between nominal and effective annual rate. If I am told that the interest rate per week is $0.001$ then I would assume this is a rate that is compounded at a weekly frequency. The nominal annual rate here is $52\times 0.001$.
– RRL
Sep 13 '20 at 14:18
• The problem in the notes says time to maturity is 9 weeks, rfi = 0.1% per week thus a 10000 means 10 usd every week. If I follow your line of thinking, this means that we have indeed $r = 0.001$, you say $R$ must be the interest rate for the single step period. We have 9 weeks left, and 3 steps is indeed 3 periods; $R = (1+r)^3 -1= (1.001^3 -1) = 0.003$, plug into q, gives us (1.003 - 0.943)/(1.0606−0.943) = 0.5102. If we plug this in your formula we get 0.9910⋅((0.1328⋅7.65)+(3⋅(0.2603⋅0.4898)1.03))= 3.46 which seems a bit high.. Sep 13 '20 at 14:55
• @wecandothis: I get $q= 0.5108$ and $q^3C_{33} + 3q^2(1-q)C_{32} = 1.4140$. Dividing by $(1+r)^9 = 1.009036$ I get $C = 1.4013$. This is the same as the result obtained by working backwards through the tree.
– RRL
Sep 15 '20 at 0:40


text:
# How to Check-Point Deep Learning Models in Keras

Last Updated on August 27, 2020

Deep learning models can take hours, days or even weeks to train.

If the run is stopped unexpectedly, you can lose a lot of work.

In this post you will discover how you can check-point your deep learning models during training in Python using the Keras library.

Kick-start your project with my new book Deep Learning With Python, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.

• Update Mar/2017: Updated for Keras 2.0.2, TensorFlow 1.0.1 and Theano 0.9.0.
• Update Sep/2019: Updated for Keras 2.2.5 API.
• Update Oct/2019: Updated for Keras 2.3.0 API.

How to Check-Point Deep Learning Models in Keras
Photo by saragoldsmith, some rights reserved.

## Checkpointing Neural Network Models

Application checkpointing is a fault tolerance technique for long running processes.

It is an approach where a snapshot of the state of the system is taken in case of system failure. If there is a problem, not all is lost. The checkpoint may be used directly, or used as the starting point for a new run, picking up where it left off.

When training deep learning models, the checkpoint is the weights of the model. These weights can be used to make predictions as is, or used as the basis for ongoing training.

The Keras library provides a checkpointing capability by a callback API.

The ModelCheckpoint callback class allows you to define where to checkpoint the model weights, how the file should named and under what circumstances to make a checkpoint of the model.

The API allows you to specify which metric to monitor, such as loss or accuracy on the training or validation dataset. You can specify whether to look for an improvement in maximizing or minimizing the score. Finally, the filename that you use to store the weights can include variables like the epoch number or metric.

The ModelCheckpoint can then be passed to the training process when calling the fit() function on the model.

Note, you may need to install the h5py library to output network weights in HDF5 format.

### Need help with Deep Learning in Python?

Take my free 2-week email course and discover MLPs, CNNs and LSTMs (with code).

Click to sign-up now and also get a free PDF Ebook version of the course.

## Checkpoint Neural Network Model Improvements

A good use of checkpointing is to output the model weights each time an improvement is observed during training.

The example below creates a small neural network for the Pima Indians onset of diabetes binary classification problem. The example assume that the pima-indians-diabetes.csv file is in your working directory.

The example uses 33% of the data for validation.

Checkpointing is setup to save the network weights only when there is an improvement in classification accuracy on the validation dataset (monitor=’val_accuracy’ and mode=’max’). The weights are stored in a file that includes the score in the filename (weights-improvement-{val_accuracy=.2f}.hdf5).

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example produces the following output (truncated for brevity).

You will see a number of files in your working directory containing the network weights in HDF5 format. For example:

This is a very simple checkpointing strategy.

It may create a lot of unnecessary check-point files if the validation accuracy moves up and down over training epochs. Nevertheless, it will ensure that you have a snapshot of the best model discovered during your run.

## Checkpoint Best Neural Network Model Only

A simpler check-point strategy is to save the model weights to the same file, if and only if the validation accuracy improves.

This can be done easily using the same code from above and changing the output filename to be fixed (not include score or epoch information).

In this case, model weights are written to the file “weights.best.hdf5” only if the classification accuracy of the model on the validation dataset improves over the best seen so far.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running this example provides the following output (truncated for brevity).

You should see the weight file in your local directory.

This is a handy checkpoint strategy to always use during your experiments.

It will ensure that your best model is saved for the run for you to use later if you wish. It avoids you needing to include code to manually keep track and serialize the best model when training.

Now that you have seen how to checkpoint your deep learning models during training, you need to review how to load and use a checkpointed model.

The checkpoint only includes the model weights. It assumes you know the network structure. This too can be serialize to file in JSON or YAML format.

In the example below, the model structure is known and the best weights are loaded from the previous experiment, stored in the working directory in the weights.best.hdf5 file.

The model is then used to make predictions on the entire dataset.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example produces the following output.

## Summary

In this post you have discovered the importance of checkpointing deep learning models for long training runs.

You learned two checkpointing strategies that you can use on your next deep learning project:

1. Checkpoint Model Improvements.
2. Checkpoint Best Model Only.

You also learned how to load a checkpointed model and make predictions.

## Develop Deep Learning Projects with Python!

#### What If You Could Develop A Network in Minutes

...with just a few lines of Python

Discover how in my new Ebook:
Deep Learning With Python

It covers end-to-end projects on topics like:
Multilayer PerceptronsConvolutional Nets and Recurrent Neural Nets, and more...

### 199 Responses to How to Check-Point Deep Learning Models in Keras

1. Gerrit Govaerts October 5, 2016 at 1:07 am #

79.56% with a 3 hidden layer architecture , 24 neurons each
Great blog , learned a lot

• Jason Brownlee October 5, 2016 at 8:31 am #

Thanks Gerrit, I’m glad you found it useful.

• ENAS March 2, 2021 at 7:54 pm #

excuse me i want to ask about what the accuracy_score in multilabel classification mean? ,and which metric i should consider in multilabel classification
thanks

• Jason Brownlee March 3, 2021 at 5:30 am #

Good question!

Not sure I have good advice for you. I have not thought deeply about the implications of accuracy for multilabel – sorry.

2. Lau MingFei October 21, 2016 at 1:02 pm #

Hi Jason, how can I checkpoint a model with my custom metric? The example codes you gave above is monitor = ‘val_acc’ , but when I replace it with monitor = my_metric , it displays the following warning message:

/usr/local/lib/python3.5/dist-packages/keras/callbacks.py:286: RuntimeWarning: Can save best model only with available, skipping.
‘skipping.’ % (self.monitor), RuntimeWarning)

So how should I do with this?

• Jason Brownlee October 22, 2016 at 6:55 am #

Great question Lau,

I have not tried to check point with a custom metric, sorry. I cannot give you good advice.

• Jakub July 26, 2019 at 11:18 pm #

Hi,
Try something like this:

custom_objects={‘my_metric’: my_metric })

• Volkan Yurtseven September 13, 2020 at 5:29 am #

3. Xu Zhang November 8, 2016 at 11:21 am #

Hi Jason,
A great post!

I saved the model and weights using callbacks, ModelCheckpoint. If I want to train it continuously from the last epoch, how to set the model.fit() command to start from the previous epoch? Sometimes, we need to change the learning rates after several epochs and to continue training from the last epoch. Your advice is highly appreciated.

• Jason Brownlee November 9, 2016 at 9:48 am #

Great question Xu Zhang,

You can load the model weights back into your network and then start a new training process.

• Lopez GG December 30, 2016 at 1:01 pm #

Thank you Jason. I ran a epoch and got the loss down to 353.6888. The session got disconnected so I used the weights as follows. However, I dont see a change in loss. I am loading the weights correctly ? Here is my code

>>> filename = “weights-improvement-01–353.6888-embedding.hdf5”
>>> model.fit(dataX_Arr, dataY_Arr, batch_size=batch_size, nb_epoch=15, callbacks=callbacks_list)
Epoch 1/15
147744/147771 [============================>.] – ETA: 1s – loss: -353.6892Epoch 00000: loss improved from inf to -353.68885, saving model to weights-

• Jason Brownlee December 31, 2016 at 7:02 am #

• Akshaya July 15, 2019 at 7:28 pm #

Hi Jason, in continuation to the point above of continuing training from the saved checkpoint, is it required that I set the random seed initially and use the same when I train the second time? What I have noticed is that, the first time I train, the loss seems to reduce. But when I load the checkpoint and continue training, the performance suddenly becomes very poor.

• Jason Brownlee July 16, 2019 at 8:14 am #

No.

You can expect variance in the model across runs, more here:
https://machinelearningmastery.com/faq/single-faq/why-do-i-get-different-results-each-time-i-run-the-code

4. Anthony November 29, 2016 at 9:58 pm #

Great post again Jason, thank you!

• Jason Brownlee November 30, 2016 at 7:56 am #

I’m glad you found it useful Anthony.

5. Nasarudin January 31, 2017 at 1:48 pm #

Hi Jason, thank you for your tutorial. I want to implement this checkpoint function in iris-flower model script but failed to do it. It keeps showing this error and I do not know how to solve it.

I put the ‘model checkpoint’ line after the ‘baseline model’ function and add ‘callbacks’

RuntimeError: Cannot clone object , as the constructor does not seem to set parameter callbacks

• Jason Brownlee February 1, 2017 at 10:41 am #

Hi Nasarudin, sorry I am not sure of the cause of this error.

I believe you cannot use callbacks like the checkpointing when using the KerasClassifier as is the case in the iris tutorial:
http://machinelearningmastery.com/multi-class-classification-tutorial-keras-deep-learning-library/

• Nasarudin February 6, 2017 at 1:50 am #

Hi Jason. Thank you for your reply. I have some dataset that looks same with the iris dataset but it is a lot bigger. I thought maybe I can use the callbacks when I train the dataset.

Since training a large dataset might taking a lot of time, do you have any suggestion to implement other function that can do checkpoint when using KerasClassifier?nk for the

By the way, you can refer to this link for the script that I wrote to implement the checkpoint function with KerasClassifier.

http://stackoverflow.com/questions/41937719/checkpoint-deep-learning-models-in-keras

Thank you.

• Jason Brownlee February 6, 2017 at 9:44 am #

Hi Nasarudin,

I would recommend using a standalone Keras model rather than the sklearn wrapper if you wish to use checkpointing.

6. Abner February 3, 2017 at 3:21 pm #

Great tutorials love your page. I got a question: I am trying to optimize a model based on val_acc using ModelCheckpoint. However, I get great results rather quickly (example: val_acc = .98). This is my max validation, therefore, the model that will be saved (save_best_only). However several epochs gave me the same max validation and the latter epochs have higher training accuracy. Example Epoch 3: acc = .70, val_acc = .98, Epoch 50: acc = 1.00, val_acc = .98. Clearly, I would want to save the second one which generalizes on the data plus shows great training. How do I do this without having to save every epoch? Basically, I want to pass a second sorting parameter to monitor (monitor=val_acc,acc).

Thanks.

• Jason Brownlee February 4, 2017 at 9:57 am #

Great question, off the cuff, I think you can pass multiple metrics for check-pointing.

Try multiple checkpoint objects?

Here’s the docs if that helps:
https://keras.io/callbacks/#modelcheckpoint

Ideally, you do want a balance between performance on the training dataset and on the validation dataset.

• Abner February 7, 2017 at 5:13 am #

Yes, it’s not letting me pass an array list or multiple parameters it’s only expecting 1 parameter base on literature, for now, I have to settle for using val_acc for bottlenecks/top layer and val_loss for the final model, though I would prefer more control. maybe I’ll ask for it in a feature request.

• Jason Brownlee February 7, 2017 at 10:23 am #

Great idea, add an issue on the Keras project.

• Damily March 22, 2017 at 10:12 pm #

Hi Abner, do you solve this problem now？

• Schmax July 29, 2019 at 8:36 am #

You could define a custom metric that encorporates both val_acc and val_loss

7. Fatma February 9, 2017 at 3:31 pm #

Hi Jason, thank you for your tutorial. I need to ask one question, if my input contains two images with different labels (the label represents the popularity of the image). I need to know how to feed this pair of images such that the first image pass through CNN1 and the second one pass through CNN2 Then I can merge them using the merge layer to classify which one is more popular than the other one. How can I use the library in order to handle the two different inputs?

• Jason Brownlee February 10, 2017 at 9:48 am #

Hi Fatma,

Perhaps you can reframe your problem to output the popularity for one image and use an “if” statement to rank the relative popularities of each image separately?

• Fatma February 10, 2017 at 1:53 pm #

I need to compare between the popularity value of the two input images such that the output will be the first image is high popular than the second image or vice versa then when I feed one test image (unlabeled) it should be compared with some baseline of the training data to compute its popularity

8. Nasarudin February 21, 2017 at 3:30 pm #

Hi Jason, great tutorial as always.

I want to ask regarding ‘validation_split’ in the script. What is the difference between this script that has ‘validation_script’ variable and the one from here which did not have ‘validation_split’ variable http://machinelearningmastery.com/tutorial-first-neural-network-python-keras/ ?

• Jason Brownlee February 22, 2017 at 9:56 am #

Great question Nasarudin.

If you have a validation_split, it will give you some feedback during training about the performance on the model on “unseen” data. When using a check-point, this can be useful as you can stop training when performance on validation data stops improving (a sign of possible overfitting).

9. jerpint March 18, 2017 at 7:46 am #

thanks!

10. Aleksandra Nabożny April 10, 2017 at 11:54 pm #

Hi Jason! Thank you very much for the tutorial!

Can you provide a sample code to visualize results as follows: plot some images from validation dataset with their 5 top label predictions?

• Jason Brownlee April 11, 2017 at 9:33 am #

Yes you could do this using the probabilistic output from the network.

Sorry, I do not have examples at this stage.

11. Pratik April 19, 2017 at 6:39 pm #

Thanks for the tutorial Jason 🙂

• Jason Brownlee April 20, 2017 at 9:23 am #

I’m glad you found it useful Pratik.

12. Krishan Chopra April 25, 2017 at 6:33 am #

Hello Sir, i want to thank you for this cool tutorial,
Currently i am checkpointing my model every 50 epochs. I also want to checkpoint any epoch whose val_acc is better but is not the 50th epoch. For ex. i have checkpointed 250th epoch, but val_acc for 282th epoch is besser and i have to save it but as i have specified the period to be 50, i cant save the 282th epoch.
Should i implement both attributes save_best_only=true and period=50 in ModelCheckPoint ?

• Jason Brownlee April 25, 2017 at 7:53 am #

Sorry, I’m not sure I follow.

You can use the checkpoint to save any improvement to the model regardless of epoch. Perhaps that would be an approach to try?

13. Zhihao Chen May 19, 2017 at 11:18 am #

Hi Jason,

I love your tutorial very much but just find a little bit confused about how to combine the check point with the cross validation. As is shown in your other tutorial, we don’t explicitly call model.fit when doing cross validation. Could you give me an example how to add check-point into this?

• Jason Brownlee May 20, 2017 at 5:33 am #

You may have to run the k-fold cross-validation process manually (e.g. loop the folds yourself).

14. chiyuan May 28, 2017 at 3:36 pm #

Love your articles. Learnt lost of things here.

• Jason Brownlee June 2, 2017 at 12:08 pm #

Thanks, I’m glad to hear that.

15. Abolfazl June 22, 2017 at 3:00 am #

Jason, your blog is amazing. Thanks for helping us out in learning this awesome field.

• Jason Brownlee June 22, 2017 at 6:11 am #

I’m glad you find it useful Abolfazl.

16. Xiufeng Yang June 29, 2017 at 3:43 pm #

Hi, great post! I want to save the training model not the validation model, how to set the parameters in checkpoint()?

• Jason Brownlee June 30, 2017 at 8:08 am #

The trained model is saved, there is no validation model. Just an evaluation of the model using validation data.

17. Mike September 19, 2017 at 5:55 pm #

Is it possible to checkpoint in between epochs?
I have an enormous dataset that takes 20hrs per epoch and it’s failed before it finished an epoch. It would be great if I could checkpoint every fifth of an epoch or so.

• Jason Brownlee September 20, 2017 at 5:55 am #

I would recommend using a data generator and then using a checkpoint after a fixed number of batches. This might work.

18. Mounir October 31, 2017 at 10:30 pm #

Hi Jason, great blog. Do you also happen to know how to save/store the epoch number of the last observed improvement during training? That would be very useful to study overfitting

• Jason Brownlee November 1, 2017 at 5:45 am #

Yes, you could add the “epoch” variable to the checkpoint filename.

For example:

19. Jes November 3, 2017 at 8:04 am #

is it possible to use Keras checkpoints together with Gridsearch? in case Gridsearch crashes?

• Jason Brownlee November 3, 2017 at 2:15 pm #

Not a good idea Jes. I’d recommend doing a custom grid search.

20. Abad December 3, 2017 at 10:10 pm #

Receiving following error:
TypeError Traceback (most recent call last)
in ()
72
73 filepath=”weights.best.hdf5″
—> 74 checkpoint = ModelCheckpoint(filepath, monitor=’val_acc’, verbose=0, save_best_only=True,node=’max’)
75 callbacks_checkpt = [checkpoint]
76

TypeError: __init__() got an unexpected keyword argument ‘node’

• Jason Brownlee December 4, 2017 at 7:47 am #

Looks like a typo.

Double check that you have copied the code exactly from the tutorial?

21. Aditya December 24, 2017 at 8:21 am #

Hi Jason!
Thanks for the informative post.

I have one question – in case of unexpected stoppage of the run, we have the best model weights for the epochs DONE SO FAR. How can we use this checkpoint as a STARTING point to continue with the remaining epochs?

• Jason Brownlee December 25, 2017 at 5:22 am #

You can load the weights, see the example in the tutorial of exactly this.

22. Liaqat Ali December 27, 2017 at 3:53 am #

Thanks for such a nice explanation. I want to ask if we are performing some experiments & want the neural network model to achieve high accuracy on for test set. Can we use this method to find the best tuned network or the highest possible accuracy.?????

• Jason Brownlee December 27, 2017 at 5:21 am #

This method can help find and save a well performing model during training.

23. davenso January 8, 2018 at 4:54 pm #

very cool.

24. davenso January 8, 2018 at 4:57 pm #

Based on this example, how long does it take typically to save or retrieve a checkpoint model?

• Jason Brownlee January 9, 2018 at 5:24 am #

Very fast, just a few seconds for large models.

• davenso January 9, 2018 at 3:43 pm #

Thanks, Jason. Could you kindly give an estimate, more than 10 secs?

• Jason Brownlee January 10, 2018 at 5:19 am #

Large models can take about a minute.

25. kszyniu January 12, 2018 at 5:59 am #

Nice post.
However, I have one question. Can I get number of epochs that model was trained for in other way than reading its filename?
Here’s my use case: upon loading my model I want to restore the training exactly from the point I saved it (by passing a value to initial_epoch argument in .fit()) for the sake of better-looking graphs in TensorBoard.
For example, I trained my model for 2 epochs (got 2 messages: “Epoch 1/5”, “Epoch 2/5”) and saved it. Now, I want to load that model and continue training from 3rd epoch (I expect getting message “Epoch 3/5” and so on).
Is there a better way than saving epochs to filename and then getting it from there (which seems kinda messy)?

• Jason Brownlee January 12, 2018 at 11:48 am #

You could read this from the filename. It’s just a string.

You could also have a callback that writes the last epoch completed to a file, and overwrite this file each time.

26. Max Jansen January 16, 2018 at 9:21 pm #

Great post and a fantastic blog! I can’t thank you enough!

• Jason Brownlee January 17, 2018 at 9:56 am #

Thanks, I’m glad to hear that.

27. Gabriel January 26, 2018 at 3:18 pm #

Hi there. Great posts! Quick question: have you run into this problem? “callbacks.py:405: RuntimeWarning: Can save best model only with val_acc available, skipping.”

Running on AWS GPU Compute instance, fyi. I am not going straight from the Keras.Sequence() model… instead, I am using the SearchGridCV as I am trying to perform some tuning tasks but want to save the best model. Any suggestions?

• Jason Brownlee January 27, 2018 at 5:53 am #

I would recommend not combining checkpointing with cross validation or grid search.

28. Rob February 28, 2018 at 2:16 am #

Hi Jason,

Thanks a lot for all your posts, really helpful. Can you explain to me why some epochs improve validation accuracy whilst previous epochs did not? If you do not use any random sampling in your dataset (e.g. no dropout), how can it be that epoch 12 increases validation score while epoch 11 does not? Aren’t they based on the same starting point (the model outputted by epoch 10)?

thanks!

• Jason Brownlee February 28, 2018 at 6:07 am #

The algorithm is stochastic where not every update improve the scores across all of the data.

This is a property of the learning algorithm, gradient descent, that is doing its best, but cannot “see” the whole state of the problem, but instead operates piece-wise and incrementally. This is a feature, not a bug. It often leads to better outcomes.

Does that help?

29. Pete March 2, 2018 at 11:50 pm #

I was curiosity what the different of these two version???
It was seemed that just filepath was different??

• Jason Brownlee March 3, 2018 at 8:11 am #

One keeps every improvement, one keeps only the best.

• Pete March 4, 2018 at 11:40 pm #

Why the filepath variable was so magic? Just change the filepath variable can make these different result. How is this achieved?

30. Pete March 3, 2018 at 1:20 am #

In the last, if I add one code model.save_weights('weights.hdf5'), what the difference of this weights and ModelCheckpoint best weight??

• Jason Brownlee March 3, 2018 at 8:18 am #

The difference is you are saving the weights from the final model after the last epoch instead of when a change in model skill has occurred during training.

31. Pete March 3, 2018 at 9:23 pm #

When I add one code model.save_weights('weights.hdf5') to save weight from the final model; and I also save ModelCheckpoint best weight, I found that these two hdf5 file were not the same. I was confusing that why the final model weight I saved wasn’t the best model weight.

• Jason Brownlee March 4, 2018 at 6:02 am #

Yes, that is the idea of checkpointing, that it the model at the last epoch may not be the best model, in fact it often is not.

Checkpointing will save the best model along the way.

Does that help? Perhaps re-read the tutorial to make this clearer?

• Pete March 4, 2018 at 11:32 pm #

Ok, thanks. In a sence, when we have used checkpoint, that is meaningless to use model.save_weights('weights.hdf5') again.

32. Kaushal Shetty March 29, 2018 at 6:19 am #

Hi Jason,
How do I checkpoint a regression model. Is my metric accuracy or mse in such a case? And what should I monitor in such case in the modelcheckpoint? I am training a time series model.

Thanks

33. Kakoli May 27, 2018 at 8:39 pm #

Hi Jason
While running for 50 epochs, I am checkpointing and saving the weights after every 5 epochs.
Now after 27th, VM disconn.
Then I reconnect and compile the model after loading the saved weights. Now if I evaluate, I shall get the score on the best weight till 27th epoch. But since only 25 epochs are considered, accuracy will not be good, right?
In that case, how do I continue the remaining 25 epochs with the saved weights?
Thanks

• Jason Brownlee May 28, 2018 at 5:57 am #

You can load a set of weights and continue training the model.

34. Rahmad ars May 30, 2018 at 7:02 am #

Hi jason, thanks for the tutorial. If i want to extract weight values of all layer from my model, how to do that? thanks

• Jason Brownlee May 30, 2018 at 3:06 pm #

I believe there is a get_weights() function on each layer.

35. michael alex June 18, 2018 at 2:23 pm #

Hi Jason,
Thanks for all the great tutorials, including this one. Is it possible to Grid Search model hyper-parameters and check-point the model at the same time? I can do either one independently, but not both at the same time. Can you show us?

Thanks,
Michael

• Jason Brownlee June 18, 2018 at 3:13 pm #

Yes, but you will have to write the for loops for the grid search yourself. sklearn won’t help.

36. Hung June 18, 2018 at 6:40 pm #

Thanks a lot Jason for excellent tutorial.

37. Adam July 21, 2018 at 12:06 am #

Where are the key values “02d” for epoch and “.2f” for val_acc coming from?

38. IGOR STAVNITSER July 23, 2018 at 11:27 am #

Is there a way to checkpoint model weights to memory instead of a file?
Also in your example you are maximizing val_acc. What are the merits of maximizing val_acc vs minimizing val_loss?
Thank you for a great post!

• Jason Brownlee July 23, 2018 at 2:25 pm #

I’m sure you could write a custom call back to do this. I don’t have an example.

You can choose what is important to you, accuracy might be more relevant to you when using the model to make predictions.

39. Abhijeet Gokar August 9, 2018 at 9:27 pm #

Thanks a lot, it is good practise to conciously design and implement checkpoints , in our model.

40. Yari August 14, 2018 at 3:19 am #

Hi Jason, thanks for the post (as well as many others I have read!).
I was wondering: after fitting with model.fit(....) and using checkpoint you will have the weights saved in an external file. But what’s the specific state of model instance after the training? Will it have the best weights or it will have the last weights calculated during the training?

So, to sum up. If I want to do a prediction on a test set immediately after the training/fitting should I load the best weights from the external file and then do the prediction or I could directly use model.predict(...) immediately after model.fit(...) ?

Thanks a lot for your support!

• Jason Brownlee August 14, 2018 at 6:24 am #

The file will have the model weights at the time of the checkpoint. The weights can be loaded and used directly for making predictions. This is the whole point of checkpointing – to save the best model found during training.

• Yari August 14, 2018 at 3:40 pm #

Hey Jason, thanks for getting back to me. Yes that was clear to me. What’s not clear is what weights model has at the end of the training.

Lets suppose I’m using

callbacks = [ EarlyStopping(patience=15, monitor='val_loss', min_delta=0, mode='min'), ModelCheckpoint('best-weights.h5', monitor='val_loss', save_best_only=True, save_weights_only=True) ]

after training is done I’ll have the best weights saved on best-weights.h5 but what are the weights stored in the model instance? If I do model.evaluate(...) (without loading best-weights.h5) will it use the best weights or just the weights corresponding to the last epoch?

• Jason Brownlee August 15, 2018 at 5:57 am #

They are the weights at the time the checkpoint is triggered. If it is triggered many times, you may have many weight files saved.

• Anupam Singh September 26, 2018 at 6:01 am #

model instance will have the weights of last epoch not the best one

41. Mohammed September 13, 2018 at 5:21 pm #

I have my owned pretrained model (.ckpt and .meta files). How to use such files to extract features of my dataset in form of matrix which rows represent samples and columns represent features?

• Jason Brownlee September 14, 2018 at 6:34 am #

Perhaps load in Python manually then try to populate a defined Keras manually?

42. chamith November 4, 2018 at 7:28 am #

Thanks a lot for the great description. But I have to clarify one thing regarding this. When I train the model using two callback functions model *ModelCheckpoint* and

• Jason Brownlee November 5, 2018 at 6:06 am #

I don’t follow, you can elaborate please?

43. Daniel Penalva November 27, 2018 at 10:23 pm #

Hi Jason,

Thanks for tutorial. Iam wondering if this approach works with GridSearch and how can i put the checkpoint to track the results. Also, iam working with Colab Research Notebook right now, is there a way to detect process interruption and use a checkpoint to save the model ?

Thank you for your help again!!

• Jason Brownlee November 28, 2018 at 7:41 am #

No, a grid search and checkpointing are at odds.

I’m not familiar with “Colab Research Notebook”, what is it?

• Daniel Penalva November 28, 2018 at 11:17 pm #

https://colab.research.google.com . Google initiative to promove notebooks with free virtual machines, kernels with GPU and TPU processing (yet experimental). But it disconnects from the kernel after a short time of no use, and the virtual machine can be unmounted after some hours. So its fundamental for deeplearning applications that you checkpoint and save the state to keep going after reconnecting.

Too bad for Grid Search. How can you do Fine Tuning (hyperparameters) and process babysitting (verify vizualization of results, possible overfits, variance-bias so on …) without checkpointing on Grid Search ?

thank you !

• Jason Brownlee November 29, 2018 at 7:41 am #

When you grid search, you want to know about what hyperparameters give good performance. The models are discarded – no checkpointing needed. Later, you can use good config to fit a new model.

• Daniel Penalva November 29, 2018 at 10:12 pm #

Yah, thats right, but still, without being able to finish without being disconnected from the server i will never know the model, so still the checkpoint comes in hand. But if you have in mind any other think to do the tuning together with checkpoints, please let us know !

Thanks ! 🙂

• Jason Brownlee November 30, 2018 at 6:33 am #

Why do you need the model?

The models during a grid search are discarded. You only need the hyperparameters of the best performing model.

• Daniel Penalva November 30, 2018 at 12:50 am #

Update
There seems to be a issue with Keras Cross Validation and Checkpointing that requires some gimmick turn-around:

https://github.com/keras-team/keras/issues/836

Cant understand why they closed the issue without a solution, just dropped my question there.

Will try to figure out how to do this …

• Daniel Penalva November 30, 2018 at 12:17 pm #

“The models during a grid search are discarded. You only need the hyperparameters of the best performing model.”

Sorry i wasnt clear, i cant run Keras on my laptop, its more than 12 years old. The only free infra i found is Colab Notebook, to fully run through all models in GridSearchCV i need to checkpoint and restart the computing since the persistence of the process in google’s virtual machine wont last few hours, sometimes less than hours.

44. William December 4, 2018 at 9:32 am #

Hi Jason

I have a question regarding compiling the model after weights are loaded.

# Compile model (required to make predictions)

When I looked at the Keras API https://keras.io/models/model/#compile the description for the compile function says “Configures the model for training”. I am confused on why we need to compile the model to make predictions?

Also, thought you might like to know that the Pima Indians onset of diabetes binary classification problem data set is no longer available.

Thanks

• Jason Brownlee December 4, 2018 at 2:35 pm #

Thanks, you might not need to compile the model after loading any longer, perhaps the API has changed.

45. kadir sharif December 25, 2018 at 6:54 pm #

i am training a model about 100 epochs.. Now suppose the electricity gone. and i have a model checkpoints that is saved in hdf5 format… and the model run 30 epochs… but i have the model checkpoints saved with val_acc monitor.

In this kind of situation how can i load the checkpoint in the same model to continue the training where it interrupted… and is it gonna continue training the model after 30 epochs… it will be a great help it you answer my questions.

• Jason Brownlee December 26, 2018 at 6:42 am #

Yes, you can continue training.

46. Riccardo January 10, 2019 at 3:06 am #

Hi Jason thanks for your tutorials, they are very helpful. I’ve implemented a model and then saved it with a checkpoint correctly. Unfortunately when I reload the model (the same structure) with the weights saved I can’t obtain the same predictions as before, there are slightly worse. My model is a simple model
model = Sequential()
The checkpoint is:
checkpoint = ModelCheckpoint(filename, monitor=’val_loss’, verbose=1, save_best_only=True, mode=’min’)
callbacks_list = [checkpoint]
model.fit(train_x, train_y, validation_split=0.15, epochs=epochs, batch_size=batch_size,
callbacks=callbacks_list, verbose=verbose)
Then i rebuild the same structure and then call:
and the predictions are a little different. Thanks in advance.

• Jason Brownlee January 10, 2019 at 7:56 am #

That is surprising, the model should make identical predictions before and after a save. Anything else would be a bug either in your code or in Keras. Try narrowing it down, here are some ideas:

• Riccardo January 10, 2019 at 7:34 pm #

Maybe it can be that when I reload the weights I reload the best result saved to file, while in the first run the weights are different at the end of model.fit()?

• Riccardo January 10, 2019 at 8:24 pm #

The way I can reload exactly the model trained is without using ModelCheckpoint(filename, monitor=’val_loss’, verbose=1, save_best_only=True, mode=’min’). I create the model, train it and then use model.save(file). Then with model.load(file) I have the same result. But here a question: it is the best model? Because now I don’t say to it monitor=’val_loss’, mode=’min’, save_best_only ecc.. I simply fit the model and then save it.

47. Jinwen Xi February 7, 2019 at 12:21 pm #

Hi, Jason,

Very helpful tutorial. Can I save the model after each mini-batch instead of each epoch?

• Jason Brownlee February 7, 2019 at 2:07 pm #

Yes, you could achieve that will a custom callback.

• Jinwen Xi February 7, 2019 at 2:28 pm #

Is there any template about how this custom callback will look like?

48. Jinwen Xi February 11, 2019 at 6:16 pm #

Thanks. I followed the instructions and created a custom callback to:

(1) at the end of each epoch, save the whole model to a file(model_pre.h5), and post-process the file and dump it to model_post.h5
(2) at the begin of next epoch, load the model from the post-processed file model_post.h5

The main part of the code implementing (1) and (2) is below:

But it seems like the model was not correctly updated using the ‘./model_post.h5’ file when I check the parameter values using HDF5View. Could you let me know if I did it in the correct way? Thanks.

-Jinwen

• Jason Brownlee February 12, 2019 at 7:55 am #

Wow!

I’m surprised that it is possible to load the model prior to each batch or epoch. Perhaps this is not working as expected?

49. Wonbin February 15, 2019 at 1:29 am #

Hi Jason, thanks for this great tutorial!
In a regression task, among ‘val_loss’ and other metrics (like ‘mse’, ‘mape’ and so on) for the monitor argument, which one would be more important to finalize the model?

Maybe I’m basically asking about the fundamental difference between loss and metrics..?
I’ve been just guessing that I might have to choose a metric not the ‘val_loss’ to see the prediction performance of the final model. Would this be correct?

• Jason Brownlee February 15, 2019 at 8:09 am #

Loss and metrics can be the same thing or different.

It comes down to what you and project stakeholders value in a final model (metric) and what must be optimized to achieve that (loss).

50. Judson February 19, 2019 at 11:40 am #

Hello Jason thanks for the posts.

as a suggestion could you show us how to checkpoint using Xgboost. Having difficulty figuring it out on my own.

51. Fredrick Ughimi February 27, 2019 at 2:51 am #

Hello Jason,

Straight as an arrow. You made it so easy to follow. This is really less abstract.

Thank you.

• Jason Brownlee February 27, 2019 at 7:34 am #

Thanks, I’m happy it helped.

• Matt July 12, 2019 at 8:35 am #

I used this post as the base for a model. I’ve been using precision and recall from the keras_metrics module and can’t seem to get it working as the monitored metric for the checkpoint function. Just tells me:

RuntimeWarning: Can save best model only when val_recall: available: skipping.

But after each epoch the val_recall is calculated and displayed so I’m not quite sure what is wrong?

• Jason Brownlee July 12, 2019 at 8:50 am #

Is “monitor” set to your custom metric?

52. Magnus Wik May 6, 2019 at 7:54 pm #

Dear Jason,

I am confused about Keras callback. You use save_best_only=True to save the weights, but according to Keras this is the setting for saving the latest best model. For weights it should be save_weights_only=True. By default, save_weights_only is set to false. Am I missing something?

Also, in one of my Jupyter notebooks, my weights are not saved at all, but in another they are, although the code is the same. It is weird.

• Magnus May 6, 2019 at 11:39 pm #

Dear Jason,

Now I understand. When you load “weights.best.hdf5”, you are actually loading the full model including the weights. So there is no need to create the model before.

In section “Loading a Check-Pointed Neural Network Model” I skipped lines 11-14 and then:

and I got the same result.

• Jason Brownlee May 7, 2019 at 6:17 am #

Nice work.

• Magnus Wik May 7, 2019 at 8:29 pm #

Thanks.
Maybe you should update the text, since now you write “The checkpoint only includes the model weights. It assumes you know the network structure.”, which is incorrect. The checkpoint includes the full model.
If only the weights are saved it should be ‘save_weights_only=True’.
What is confusing is that it is possible to treat the full model files as weights, using model.load_weights. Personally I think it should throw a warning.

• Jason Brownlee May 7, 2019 at 6:16 am #

Perhaps try running the example from the command line?

53. Steven Gonzalez May 17, 2019 at 11:45 am #

Thanks for a great tutorial!

• Jason Brownlee May 17, 2019 at 2:52 pm #

You’re welcome, I’m glad it helped.

54. hayj May 27, 2019 at 12:56 am #

Hello Jason,

Great tutorial again!
I wanted to ask you how to monitor my val_top_k_categorical_accuracy instead of val_acc ?
I can’t find anything about it on internet.
I tryed differents things change the position of my metrics metrics=['top_k_categorical_accuracy', 'accuracy'], try monitor="top_k_categorical_accuracy"… but nothing works

• Jason Brownlee May 27, 2019 at 6:51 am #

Interesting. If you add the metric to the list of metrics does it appear in the history dict?

If so, you can use that name.

55. hayj May 28, 2019 at 1:37 am #

Anyway I solved the problem by defining my own callback which save a model when I observe any “val_*” improvement

• Jason Brownlee May 28, 2019 at 8:19 am #

Nicely done, I would have done the same 🙂

56. Dang Nguyen Hong June 27, 2019 at 6:23 am #

Each time i search for the answer of a question, your blog solves it ! Great work, thanks so much!

57. zeinab July 22, 2019 at 5:08 am #

As usual, great tutorial

58. alilouche August 6, 2019 at 8:10 am #

I can’t save my model using your instructions, it does not inform me an error but does not register

• Jason Brownlee August 6, 2019 at 2:00 pm #

Sorry to hear that. Perhaps try reducing your example to the simplest possible code example?

Perhaps try posting your code and issue to stackoverflow?

59. zeinab August 13, 2019 at 12:40 pm #

How can I load weights when I use a custom metrics?

• Jason Brownlee August 13, 2019 at 2:37 pm #

As follows:

• zeinab August 14, 2019 at 6:55 pm #

Thank you, Jason.

model.load_weights("bestweights.hdf5")
unfortunately, this function, cannot see the custom metric.

• Jason Brownlee August 15, 2019 at 7:59 am #

Yes, you must define the metric and load the weights in the same scope.

• zeinab August 15, 2019 at 8:13 am #

sorry, but I donot understand what do you mean?

60. zeinab August 14, 2019 at 9:32 pm #

For a certain problem, I try to solve it using CNN model on 5 cross validation, I find its performance through the average of the loss of the 5 folds and the average of the 5 folds accuracy. (I stop training when i reach the minimum validation loss)
1- Is this is a right way for calculating a model performance?

Then I evaluate the same problem on the same dataset but using LSTM.

2- Now, I want to select the best model, How?

3- Does the best model is the model with the lowest loss or the highest accuracy or what?

61. zeinab August 15, 2019 at 2:43 am #

My problem is regression not classification?

62. zeinab August 15, 2019 at 8:19 am #

What about choosing the best model (CNN, LSTM, …) applied on the same problem and the same dataset?

63. zeinab August 15, 2019 at 8:35 am #

I apply more than one algorithm(CNN, LSTM) to solve my problem (text similarity). How can I select the best algorithm?

Does the best algorithm is the algorithm that has the lowest validation loss?

• Jason Brownlee August 15, 2019 at 2:18 pm #

It is common to choose a model based both on complexity (minimized) and on skill (optimizing a domain-specific metric).

Loss is a good proxy for the domain specific metric, but hard to communicate to subject matter experts/stakeholders.

64. ylnhari August 15, 2019 at 9:27 pm #

Hi,

If i want to check point both the best model and model at last epoch when the training halted in middle because of some other reason , how to do that ?

• Jason Brownlee August 16, 2019 at 7:51 am #

Perhaps configure a different ModelCheckpoint instance for each case?

65. kumar August 16, 2019 at 9:12 pm #

Dear Jason,

We are using the following statements to save the model.

filepath=”model_lstm_10M_{epoch:02d}.h5″
checkpoint = ModelCheckpoint(filepath, period=1000, verbose=1, save_best_only=False)
#tbCallBack = TensorBoard(log_dir=’./log’, histogram_freq=0, write_graph=True, write_images=True)
callbacks_list = [checkpoint]

# fit model
model.fit(train_data, target, epochs=10000,batch_size=4,callbacks=callbacks_list,verbose=2)
model.save(‘model_10M_lstm_100.h5’)

We stop training at 2300 epochs, when we start again, it starts from epoch 1 I want to continue from previous epochs (2300) . What are the changes we have to do to achieve this?

• Jason Brownlee August 17, 2019 at 5:42 am #

Training will always start at epoch 0.

If you load the model and start training again, it will start with weights you from the end of the last run. Only the epoch number will reset, not the weights.

If you know how many epochs were completed, you can subtract that from the number of epochs you wish to use in the second run.

66. Jacky QIN October 17, 2019 at 1:47 pm #

Hi, Jason. Your course is pretty good, I get a lot. But there is a little bug, I guess that’s the API have been updated. The code:

checkpoint = ModelCheckpoint(filepath, monitor=’val_accuracy’, verbose=1, save_best_only=True, mode=’max’)

The argument ‘monitor’ should be ‘val_acc’ not ‘val_accuracy’

• Jason Brownlee October 17, 2019 at 1:54 pm #

The examples assume Keras 2.3 or higher where you must use val_accuracy.

For older versions of Keras, you must use val_acc.

67. Eduardo October 28, 2019 at 2:32 am #

Hi Jason, do you know of a way to also save the epoch number of the last checkpointed model?

What I want to do is, after training, evaluate the difference between Train and Validation loss of the training epoch corresponding to the “best checkpointed model” to have a reference of the error gap at that exact point.

Right now I’m calculating that “error gap” from the results of the model validation: train loss vs test loss.

• Jason Brownlee October 28, 2019 at 6:08 am #

You can save it into the filename of the model

68. Walid November 7, 2019 at 3:03 am #

Great clear post
is not ModelCheckpoint saving the full model with weights?

I think “save_weights_only” is by default false

69. Yogeeshwari February 11, 2020 at 11:06 pm #

I am trying to save my model after every epoch. I am monitoring for accuracy. After first epoch, I get an OS error stating “Unable to open file: name = /logs/weights-improvement-01-0.55.hdf5′, errno = 2, error message = ‘No such file or directory’, flags = 13, o_flags = 242)”. Kindly let me know.

• Jason Brownlee February 12, 2020 at 5:47 am #

Perhaps try saving to a different location, e.g. /tmp/

70. David April 16, 2020 at 8:44 pm #

Hey. Great blog. I wanted to ask how can I do something (e.g save an image) when it is best epoch.

• Jason Brownlee April 17, 2020 at 6:19 am #

Thanks!

You could create your own call back, and perform any action you like on any condition you like.

71. Yunes April 22, 2020 at 2:43 pm #

Hi Jason! Thank you very much for the tutorial!
I have a doubt, selecting the model weights that get the best performance over the validation set (monitor = val_accuracy, mode = ‘max’), wouldn’t that be a somewhat optimistic result?
On the other hand, will the control points always be applied based on the validation set?

• Jason Brownlee April 23, 2020 at 5:56 am #

It could be if the validation dataset is small or not representative.

72. Maria June 24, 2020 at 3:42 am #

Hi Jason, and thank you for your contribution.

I just need to clarify something.

After designing a DNN, we
1. compile()
2. fit()
3. evaluate()

If the model at the last epoch is not the one with the best weights, why we do not always use callback to save the best model, and evaluate with those ones:
1. compile()
2.fit() with callback to save the best weights
4. compile()
5. evaluate()

73. sei September 18, 2020 at 8:50 pm #

Hii @Jason …during training does the model improved or only weights…?

• Jason Brownlee September 19, 2020 at 6:53 am #

The weights are the model. Training changes the weights.

74. Antonio February 18, 2021 at 9:22 am #

Hi Jason,

Nice tutorial, bu I can’t run it.
I get the following error in line #15 model.add(Dense(12, input_dim=8, activation=’relu’))

module ‘tensorflow.python.framework.ops’ has no attribute ‘_TensorLike’
File “/Users/mact/Projects/src/test.py”, line 18, in get_model

75. tiennguyen February 20, 2021 at 1:53 pm #

Thank for sharing Jason Brownlee.
I have a question. Currently, checkpoint saved every each epochs and improvement, If I want to save after N epochs or N batch_size how do I have to do. Can you suggest me an idea? thanks a lot.

• tiennguyen February 20, 2021 at 2:07 pm #

And I see the parameter save_weights_only = False, so the last example of topic, you do not have to create model by some following commands: # create model
model = Sequential()


text:
# Number in Characters in C Sharp

## Number in Characters

To convert a number in characters in C#, we can use a loop and switch case. After taking input from the user, the number is iterated and is divided by 10 until it becomes 0. The remainder so obtained is passed in switch case to retrieve the word for the number.

Example:

using System; public class Example { public static void Main(string[] args) { int num, sum=0, x; Console.Write("Enter a Number: "); num = int.Parse(Console.ReadLine()); while(num>0) { x = num%10; sum = sum*10 + x; num = num/10; } num = sum; while(num>0) { x = num%10; switch(x) { case 1: Console.Write("One "); break; case 2: Console.Write("Two "); break; case 3: Console.Write("Three "); break; case 4: Console.Write("Four "); break; case 5: Console.Write("Five "); break; case 6: Console.Write("Six "); break; case 7: Console.Write("Seven "); break; case 8: Console.Write("Eight "); break; case 9: Console.Write("Nine "); break; case 0: Console.Write("Zero "); break; default: Console.Write("Not_a_Digit "); break; } num = num/10; } } }

Output:

Explanation:

In the above example, we are converting a number in characters.


text:
# Inflation of the Universe as Bose Einstein Condensate (Literature)

I'm searching for literature in the field of cosmological models which use a Bose-Einstein-Condensate (BEC) to model the process of inflation. Especially the analytical results the cubic BEC models are interesting for me.

Additional if you can recommend any papers which do numerics for BEC are of interest too. I also try to understand the relativistic correction of a BEC of a scalar field $\psi$ which dynamic is determined by the Gross-Pitaevskii equation. As far as I know this adapted equation has to be of Klein-Gordon type $$\left( \Box +m^2 \right) \phi = - \frac{\lambda}{3!} \left( \phi^{+} \phi\right) \phi,$$ where $\lambda$ is a suitable constant. So it would be nice if you can recommend some good papers on this to

Before answering, please see our policy on resource recommendation questions. Please write substantial answers that detail the style, content, and prerequisites of the book, paper or other resource. Explain the nature of the resource so that readers can decide which one is best suited for them rather than relying on the opinions of others. Answers containing only a reference to a book or paper will be removed!

• Why do think BECs can be used to model inflation? What's the equation of state of the BEC? – bapowell Aug 3 '18 at 14:31


text:
# EU parliament OKs no-action powers but leaked doc signals delay

## Council note means regulators may not get no-action powers until at least November

The European Parliament has voted to grant pan-European financial watchdogs the power to postpone incoming rules in some cases. But a leaked document from another legislative body suggests the regulators may not be able to exercise their new powers until at least November.

For financial firms, this is good news and bad news. They will welcome the major step towards a formal no-action procedure in the European Union, already long in existence in the US. They may, though, have to wait a while yet before the three European Supervisory Authorities (ESAs) can reliably delay or suspend the enforcement of potentially damaging regulation.

“The parliament’s approval of no-action letters is an important development and it means they will now definitely be on the agenda in the trilogue [legislative] negotiations,” says Pedro Pinto, a director of advocacy at the Association for Financial Markets in Europe (Afme). “The letters are needed because the current way the ESAs address these issues doesn’t provide relief nor sufficient legal certainty.”

However, the final go-ahead from all three EU law-making bodies, including the European Commission, could be delayed – because of the Council of the European Union. A January 11 note from Romania, which holds the presidency of the council this year, recommends prioritising work on other reforms over the coming two months.

This makes it unlikely the council will finalise its position on no-action relief in time for the subsequent trilogue talks to conclude before the parliament breaks in April for elections in May.

All legislative work by the EU will then be suspended until a new commission is formed, with approval from the new parliament. The multi-step process dictates that a new commission will be formed in October or November, with the first trilogue completed the following month at the earliest.

### Parliament’s say

On January 10, the European Parliament approved its own version of a no-action mechanism, overcoming internal disagreements over what form it should take and whether the ESAs should even be given no-action powers in the first place.

The parliament’s text allows the watchdogs to postpone both primary and secondary legislation. Financial firms had advocated exactly such broad powers, saying that sometimes the rule from which they needed relief was contained in a level one law.

In the parliament’s iteration, the ESAs will be able to delay the enactment of a new rule in three scenarios: if doing so put market participants in breach of other EU regulatory obligations; if the authorities needed to clarify the rule; or if compliance posed a threat to market confidence, customer or investor protection, the functioning of financial or commodity markets, or the stability of the EU financial system.

The text says the European Commission, the parliament or the council may ask the authority to reconsider its decision to issue a no-action letter.

There is also a time limit on the relief: it can last up to six months and can be renewed only once for another six months.