text:
# An Event-Driven Web Framework

Leo (better known online as inaimathi) is a recovering Graphic Designer who has professionally written Scheme, Common Lisp, Erlang, Javascript, Haskell, Clojure, Go, Python, PHP and C. He currently blogs about programming, plays board games and works at a Ruby-based startup in Toronto, Ontario.

In 2013, I decided to write a web-based game prototyping tool for card and board games called House. In these types of games, it is common for one player to wait for another player to make a move; however, when the other player finally does take action, we would like for the waiting player to be notified of the move quickly thereafter.

This is a problem that turns out to be more complicated than it first seems. In this chapter, we'll explore the issues with using HTTP to build this sort of interaction, and then we'll build a web framework in Common Lisp that allows us to solve similar problems in the future.

## The Basics of HTTP Servers

At the simplest level, an HTTP exchange is a single request followed by a single response. A client sends a request, which includes a resource identifier, an HTTP version tag, some headers and some parameters. The server parses that request, figures out what to do about it, and sends a response which includes the same HTTP version tag, a response code, some headers and a response body.

Notice that, in this description, the server responds to a request from a specific client. In our case, we want each player to be updated about any moves as soon as they happen, rather than only getting notifications when their own move is made. This means we need the server to push messages to clients without first receiving a request for the information.1

There are several standard approaches to enabling server push over HTTP.

### Comet/Long Poll

The "long poll" technique has the client send the server a new request as soon as it receives a response. Instead of fulfilling that request right away, the server waits on a subsequent event to respond. This is a bit of a semantic distinction, since the client is still taking action on every update.

### Server-Sent Events (SSE)

Server-sent events require that the client initiates a connection and then keeps it open. The server periodically writes new data to the connection without closing it, and the client interprets incoming new messages as they arrive rather than waiting for the response connection to terminate. This is a bit more efficient than the Comet/long poll approach because each message doesn't have to incur the overhead of new HTTP headers.

### WebSockets

WebSockets are a communication protocol built on top of HTTP. The server and client open up an HTTP conversation, then perform a handshake and protocol escalation. The end result is that they're still communicating over TCP/IP, but they're not using HTTP to do it at all. The advantage this has over SSEs is that you can customize the protocol for efficiency.

### Long-Lived Connections

These three approaches are quite different from one another, but they all share an important characteristic: they all depend on long-lived connections. Long polling depends on the server keeping requests around until new data is available, SSEs keep an open stream between client and server to which data is periodically written, and WebSockets change the protocol a particular connection is using, but leave it open.

To see why this might cause problems for your average HTTP server, let's consider how the underlying implementation might work.

### Traditional HTTP Server Architecture

A single HTTP server processes many requests concurrently. Historically, many HTTP servers have used a thread-per-request architecture. That is, for each incoming request, the server creates a thread to do the work necessary to respond.

Since each of these connections is intended to be short-lived, we don't need many threads executing in parallel to handle them all. This model also simplifies the implementation of the server by enabling the server programmer to write code as if there were only one connection being handled at any given time. It also gives us the freedom to clean up failed or "zombie" connections and their associated resources by killing the corresponding thread and letting the garbage collector do its job.

The key observation is that an HTTP server hosting a "traditional" web application that has $N$ concurrent users might only need to handle a very small fraction of $N$ requests in parallel to succeed. For the type of interactive application that we are trying to build, $N$ users will almost certainly require the application to maintain at least $N$ connections in parallel, at once.

The consequence of keeping long-lived connections around is that we'll need either:

• A platform where threads are "cheap" enough that we can use large numbers of them at once.
• A server architecture that can handle many connections with a single thread.

There are programming environments such as Racket, Erlang, and Haskell that provide thread-like constructs that are "lightweight" enough to consider the first option. This approach requires the programmer to explicitly deal with synchronization issues, which are going to be much more prevalent in a system where connections are open for a long time and likely all competing for similar resources. Specifically, if we have some sort of central data shared by several users simultaneously, we will need to coordinate reads and writes of that data in some way.

If we don't have cheap threads at our disposal or we are unwilling to work with explicit synchronization, we must consider having a single thread handle many connections.2 In this model, our single thread is going to be handling tiny "slices" of many requests all at once, switching between them as efficiently as it possibly can. This system architecture pattern is most commonly referred to as event-driven or event-based.3

Since we are only managing a single thread, we don't have to worry as much about protecting shared resources from simultaneous access. However, we do have a unique problem of our own in this model. Since our single thread is working on all in-flight requests at once, we must make sure that it never blocks. Blocking on any connection blocks the entire server from making progress on any other request. We have to be able to move on to another client if the current one can't be serviced further, and we need to be able to do so in a manner that doesn't throw out the work done so far.4

While it is uncommon for a programmer to explicitly tell a thread to stop working, many common operations carry a risk of blocking. Because threads are so prevalent, and reasoning about asynchronicity is a heavy burden on the programmer, many languages and their frameworks assume that blocking on I/O is a desirable property. This makes it very easy to block somewhere by accident. Luckily, Common Lisp does provide us with a minimal set of asynchronous I/O primitives which we can build on top of.

### Architectural Decisions

Now that we've studied the background of this problem, we've arrived at the point where we need to make informed decisions about what we are building.

At the time I started thinking about this project, Common Lisp didn't have a complete green-thread implementation, and the standard portable threading library doesn't qualify as "really REALLY cheap". The options amounted to either picking a different language, or building an event-driven web server for my purpose. I chose the latter.

In addition to the server architecture, we also need to choose which of the three server-push approaches to use. The use-case we are considering (an interactive multiplayer board game) requires frequent updates to each client, but relatively sparse requests from each client, which fits the SSE approach to pushing updates, so we'll go with this.

Now that we've motivated our architectural decision and decided on a mechanism for simulating bidirectional communication between clients and server, let's get started on building our web framework. We'll start by building a relatively "dumb" server first, and then we'll extend it into a web-application framework that lets us focus on what our heavily-interactive program needs to do, and not how it is doing it.

## Building an Event-Driven Web Server

Most programs that use a single process to manage concurrent streams of work use a pattern called an event loop. Let's look at what an event loop for our web server might look like.

### The Event Loop

Our event loop needs to:

• listen for incoming connections;
• handle all new handshakes or incoming data on existing connections;
• clean up dangling sockets that are unexpectedly killed (e.g. by an interrupt)
(defmethod start ((port integer))
(let ((server (socket-listen
usocket:*wildcard-host* port
:reuse-address t
:element-type 'octet))
(conns (make-hash-table)))
(unwind-protect
(loop (loop for ready
in (wait-for-input
(cons server (alexandria:hash-table-keys conns))
:ready-only t)
do (process-ready ready conns)))
(loop for c being the hash-keys of conns
do (loop while (socket-close c)))
(loop while (socket-close server)))))

If you haven't written a Common Lisp program before, this code block requires some explanation. What we have written here is a method definition. While Lisp is popularly known as a functional language, it also has its own system for object-oriented programming called "The Common Lisp Object System", which is usually abbreviated as "CLOS".5

### CLOS and Generic Functions

In CLOS, instead of focusing on classes and methods, we write generic functions that are implemented as collections of methods. In this model, methods don't belong to classes, they specialize on types.6 The start method we just wrote is a unary method where the argument port is specialized on the type integer. This means that we could have several implementations of start where port varies in type, and the runtime will select which implementation to use depending on the type of port when start is called.

More generally, methods can specialize on more than one argument. When a method is called, the runtime:

• dispatches on the type of its arguments to figure out which method body should be run, and
• runs the appropriate function.

### Processing Sockets

We'll see another generic function at work in process-ready, which was called earlier from our event loop. It processes a ready socket with one of two methods, depending on the type of socket we are handling.

The two types we're concerned with are the stream-usocket, which represents a client socket that will make a request and expect to be sent some data back, and the stream-server-usocket, which represents our local TCP listener that will have new client connections for us to deal with.

If a stream-server-socket is ready, that means there's a new client socket waiting to start a conversation. We call socket-accept to accept the connection, and then put the result in our connection table so that our event loop can begin processing it with the others.

(defmethod process-ready ((ready stream-server-usocket) (conns hash-table))
(setf (gethash (socket-accept ready :element-type 'octet) conns) nil))

When a stream-usocket is ready, that means that it has some bytes ready for us to read. (It's also possible that the other party has terminated the connection.)

(defmethod process-ready ((ready stream-usocket) (conns hash-table))
(let ((buf (or (gethash ready conns)
(setf (gethash ready conns)
(make-instance 'buffer :bi-stream (flex-stream ready))))))
(if (eq :eof (buffer! buf))
(ignore-errors
(remhash ready conns)
(socket-close ready))
(let ((too-big?
(> (total-buffered buf)
+max-request-size+))
(too-old?
(> (- (get-universal-time) (started buf))
+max-request-age+))
(too-needy?
(> (tries buf)
+max-buffer-tries+)))
(cond (too-big?
(error! +413+ ready)
(remhash ready conns))
((or too-old? too-needy?)
(error! +400+ ready)
(remhash ready conns))
((and (request buf) (zerop (expecting buf)))
(remhash ready conns)
(when (contents buf)
(setf (parameters (request buf))
(nconc (parse buf) (parameters (request buf)))))
(handler-case
(handle-request ready (request buf))
(http-assertion-error () (error! +400+ ready))
((and (not warning)
(not simple-error)) (e)
(error! +500+ ready e))))
(t
(setf (contents buf) nil)))))))

This is more involved than the first case. We:

1. Get the buffer associated with this socket, or create it if it doesn't exist yet;
2. Read output into that buffer, which happens in the call to buffer!;
3. If that read got us an :eof, the other side hung up, so we discard the socket and its buffer;
4. Otherwise, we check if the buffer is one of complete?, too-big?, too-old? or too-needy?. If so, we remove it from the connections table and return the appropriate HTTP response.

This is the first time we're seeing I/O in our event loop. In our discussion in Traditional HTTP Server Architecture, we mentioned that we have to be very careful about I/O in an event-driven system, because we could accidentally block our single thread. So, what do we do here to ensure that this doesn't happen? We have to explore our implementation of buffer! to find out exactly how this works.

### Processing Connections Without Blocking

The basis of our approach to processing connections without blocking is the library function read-char-no-hang, which immediately returns nil when called on a stream that has no available data. Where there is data to be read, we use a buffer to store intermediate input for this connection.

(defmethod buffer! ((buffer buffer))
(handler-case
(let ((stream (bi-stream buffer)))
(incf (tries buffer))
(loop for char = (read-char-no-hang stream) until (null char)
do (push char (contents buffer))
do (incf (total-buffered buffer))
when (request buffer) do (decf (expecting buffer))
when (line-terminated? (contents buffer))
do (multiple-value-bind (parsed expecting) (parse buffer)
(setf (request buffer) parsed
(expecting buffer) expecting)
(return char))
when (> (total-buffered buffer) +max-request-size+) return char
finally (return char)))
(error () :eof)))

When buffer! is called on a buffer, it:

• increments the tries count, so that we can evict "needy" buffers in process-ready;
• loops to read characters from the input stream, and
• returns the last character it read if it has read all of the available input.

It also tracks any \r\n\r\n sequences so that we can later detect complete requests. Finally, if any error results, it returns an :eof to signal that process-ready should discard this connection.

The buffer type is a CLOS class. Classes in CLOS let us define a type with fields called slots. We don't see the behaviours associated with buffer on the class definition, because (as we've already learned), we do that using generic functions like buffer!.

defclass does allow us to specify getters/setters (readers/accessors), and slot initializers; :initform specifies a default value, while :initarg identifies a hook that the caller of make-instance can use to provide a default value.

(defclass buffer ()
((tries :accessor tries :initform 0)
(contents :accessor contents :initform nil)
(bi-stream :reader bi-stream :initarg :bi-stream)
(total-buffered :accessor total-buffered :initform 0)
(started :reader started :initform (get-universal-time))
(request :accessor request :initform nil)
(expecting :accessor expecting :initform 0)))

Our buffer class has seven slots:

• tries, which keeps count of how many times we've tried reading into this buffer
• contents, which contains what we've read so far
• bi-stream, which a hack around some of those Common Lisp-specific, non-blocking-I/O annoyances I mentioned earlier
• total-buffered, which is a count of chars we've read so far
• started, which is a timestamp that tells us when we created this buffer
• request, which will eventually contain the request we construct from buffered data
• expecting, which will signal how many more chars we're expecting (if any) after we buffer the request headers

### Interpreting Requests

Now that we've seen how we incrementally assemble full requests from bits of data that are pooled into our buffers, what happens when we have a full request ready for handling? This happens in the method handle-request.

(defmethod handle-request ((socket usocket) (req request))
(aif (lookup (resource req) *handlers*)
(funcall it socket (parameters req))
(error! +404+ socket)))

This method adds another layer of error handling so that if the request is old, big, or needy, we can send a 400 response to indicate that the client provided us with some bad or slow data. However, if any other error happens here, it's because the programer made a mistake defining a handler, which should be treated as a 500 error. This will inform the client that something went wrong on the server as a result of their legitimate request.

If the request is well-formed, we do the tiny and obvious job of looking up the requested resource in the *handlers* table. If we find one, we funcall it, passing along the client socket as well as the parsed request parameters. If there's no matching handler in the *handlers* table, we instead send along a 404 error. The handler system will be part of our full-fledged web framework, which we'll discuss in a later section.

We still haven't seen how requests are parsed and interpreted from one of our buffers, though. Let's look at that next:

(defmethod parse ((buf buffer))
(let ((str (coerce (reverse (contents buf)) 'string)))
(if (request buf)
(parse-params str)
(parse str))))

This high-level method delegates to a specialization of parse that works with plain strings, or to parse-params that interprets the buffer contents as HTTP parameters. These are called depending on how much of the request we've already processed; the final parse happens when we already have a partial request saved in the buffer, at which point we're only looking to parse the request body.

(defmethod parse ((str string))
(let ((lines (split "\\r?\\n" str)))
(destructuring-bind (req-type path http-version) (split " " (pop lines))
(declare (ignore req-type))
(assert-http (string= http-version "HTTP/1.1"))
(let* ((path-pieces (split "\\?" path))
(resource (first path-pieces))
(parameters (second path-pieces))
(req (make-instance 'request :resource resource)))
(loop
for header = (pop lines)
for (name value) = (split ": " header)
until (null name)
do (push (cons (->keyword name) value) (headers req)))
(setf (parameters req) (parse-params parameters))
req))))

(defmethod parse-params ((params null)) nil)

(defmethod parse-params ((params string))
(loop for pair in (split "&" params)
for (name val) = (split "=" pair)
collect (cons (->keyword name) (or val ""))))

In the parse method specializing on string, we transform the content into usable pieces. We do so on strings instead of working directly with buffers because this makes it easier to test the actual parsing code in an environment like an interpreter or REPL.

The parsing process is:

1. Split on "\\r?\\n".
2. Split the first line of that on " " to get the request type (POST, GET, etc)/URI path/http-version.
3. Assert that we're dealing with an HTTP/1.1 request.
4. Split the URI path on "?", which gives us plain resource separate from any GET parameters.
5. Make a new request instance with the resource in place.
6. Populate that request instance with each split header line.
7. Set that requests parameters to the result of parsing our GET parameters.

As you might expect by now, request is an instance of a CLOS class:

    (defclass request ()
((resource :accessor resource :initarg :resource)
(headers :accessor headers :initarg :headers :initform nil)
(parameters :accessor parameters :initarg :parameters :initform nil)))

We've now seen how our clients can send requests and have them interpreted and handled by our server. The last thing we have to implement as part of our core server interface is the capability to write responses back to the client.

### Rendering Responses

Before we discuss rendering responses, we have to consider that there are two kinds of responses that we may be returning to our clients. The first is a "normal" HTTP response, complete with HTTP headers and body. We represent these kinds of responses with instances of the response class:

(defclass response ()
((content-type
:accessor content-type :initform "text/html" :initarg :content-type)
(charset
:accessor charset :initform "utf-8")
(response-code
:accessor response-code :initform "200 OK" :initarg :response-code)
(keep-alive?
:accessor keep-alive? :initform nil :initarg :keep-alive?)
(body
:accessor body :initform nil :initarg :body)))

The second is an SSE message, which we will use to send an incremental update to our clients.

(defclass sse ()
((id :reader id :initarg :id :initform nil)
(event :reader event :initarg :event :initform nil)
(retry :reader retry :initarg :retry :initform nil)
(data :reader data :initarg :data)))

We'll send an HTTP response whenever we receive a full HTTP request; however, how do we know when and where to send SSE messages without an originating client request?

A simple solution is to register channels7, to which we'll subscribe sockets as necessary.

(defparameter *channels* (make-hash-table))

(defmethod subscribe! ((channel symbol) (sock usocket))
(push sock (gethash channel *channels*))
nil)

We can then publish! notifications to said channels as soon as they become available.

(defmethod publish! ((channel symbol) (message string))
(awhen (gethash channel *channels*)
(setf (gethash channel *channels*)
(loop with msg = (make-instance 'sse :data message)
for sock in it
when (ignore-errors
(write! msg sock)
(force-output (socket-stream sock))
sock)
collect it))))

In publish!, we call write! to actually write an sse to a socket. We'll also need a specialization of write! on responses to write full HTTP responses as well. Let's handle the HTTP case first.

(defmethod write! ((res response) (socket usocket))
(handler-case
(with-timeout (.2)
(let ((stream (flex-stream socket)))
(flet ((write-ln (&rest sequences)
(mapc (lambda (seq) (write-sequence seq stream)) sequences)
(crlf stream)))
(write-ln "HTTP/1.1 " (response-code res))
(write-ln
"Content-Type: " (content-type res) "; charset=" (charset res))
(write-ln "Cache-Control: no-cache, no-store, must-revalidate")
(when (keep-alive? res)
(write-ln "Connection: keep-alive")
(write-ln "Expires: Thu, 01 Jan 1970 00:00:01 GMT"))
(awhen (body res)
(write-ln "Content-Length: " (write-to-string (length it)))
(crlf stream)
(write-ln it))
(values))))
(trivial-timeout:timeout-error ()
(values))))

This version of write! takes a response and a usocket named sock, and writes content to a stream provided by sock. We locally define the function write-ln which takes some number of sequences, and writes them out to the stream followed by a crlf. This is for readability; we could instead have called write-sequence/crlf directly.

Note that we're doing the "Must not block" thing again. While writes are likely to be buffered and are at lower risk of blocking than reads, we still don't want our server to grind to a halt if something goes wrong here. If the write takes more than 0.2 seconds8, we just move on (throwing out the current socket) rather than waiting any longer.

Writing an SSE out is conceptually similar to writing out a response:

(defmethod write! ((res sse) (socket usocket))
(let ((stream (flex-stream socket)))
(handler-case
(with-timeout (.2)
(format
stream "~@[id: ~a~%~]~@[event: ~a~%~]~@[retry: ~a~%~]data: ~a~%~%"
(id res) (event res) (retry res) (data res)))
(trivial-timeout:timeout-error ()
(values)))))

This is simpler than working with full HTTP responses since the SSE message standard doesn't specify CRLF line-endings, so we can get away with a single format call. The ~@[...~] blocks are conditional directives, which allow us to gracefully handle nil slots. For example, if (id res) is non-nil, we'll output id: <the id here>, otherwise we will ignore the directive entirely. The payload of our incremental update data is the only required slot of sse, so we can include it without worrying about it being nil. And again, we're not waiting around for too long. After 0.2 seconds, we'll time out and move on to the next thing if the write hasn't completed by then.

### Error Responses

Our treatment of the request/response cycle so far hasn't covered what happens when something goes wrong. Specifically, we used the error! function in handle-request and process-ready without describing what it does.

(define-condition http-assertion-error (error)
((assertion :initarg :assertion :initform nil :reader assertion))
(:report (lambda (condition stream)
(format stream "Failed assertions '~s'"
(assertion condition)))))

define-condition creates new error classes in Common Lisp. In this case, we are defining an HTTP assertion error, and stating that it will specifically need to know the actual assertion it's acting on, and a way to output itself to a stream. In other languages, you'd call this a method. Here, it's a function that happens to be the slot value of a class.

How do we represent errors to the client? Let's define the 4xx and 5xx-class HTTP errors that we'll be using often:

(defparameter +404+
(make-instance
'response :response-code "404 Not Found"
:content-type "text/plain"
:body "Resource not found..."))

(defparameter +400+
(make-instance
'response :response-code "400 Bad Request"
:content-type "text/plain"
:body "Malformed, or slow HTTP request..."))

(defparameter +413+
(make-instance
'response :response-code "413 Request Entity Too Large"
:content-type "text/plain"
:body "Your request is too long..."))

(defparameter +500+
(make-instance
'response :response-code "500 Internal Server Error"
:content-type "text/plain"
:body "Something went wrong on our end..."))

Now we can see what error! does:

(defmethod error! ((err response) (sock usocket) &optional instance)
(declare (ignorable instance))
(ignore-errors
(write! err sock)
(socket-close sock)))

It takes an error response and a socket, writes the response to the socket and closes it (ignoring errors, in case the other end has already disconnected). The instance argument here is for logging/debugging purposes.

And with that, we have an event-driven web server that can respond to HTTP requests or send SSE messages, complete with error handling!

## Extending the Server Into a Web Framework

We have now built a reasonably functional web server that will move requests, responses, and messages to and from clients. The actual work of any web application hosted by this server is done by delegating to handler functions, which were introduced in Interpreting Requests but left underspecified.

The interface between our server and the hosted application is an important one, because it dictates how easily application programmers can work with our infrastructure. Ideally, our handler interface would map parameters from a request to a function that does the real work:

(define-handler (source :is-stream? nil) (room)
(subscribe! (intern room :keyword) sock))

(define-handler (send-message) (room name message)
(publish! (intern room :keyword)
(encode-json-to-string
((:name . ,name) (:message . ,message)))))

(define-handler (index) ()
(with-html-output-to-string (s nil :prologue t :indent t)
(:html
(:head (:script
:type "text/javascript"
:src "/static/js/interface.js"))
(:body (:div :id "messages")
(:textarea :id "input")
(:button :id "send" "Send")))))

One of the concerns I had in mind when writing House was that, like any application open to the greater internet, it would be processing requests from untrusted clients. It would be nice to be able to say specifically what type of data each request should contain by providing a small schema that describes the data. Our previous list of handlers would then look like this:

(defun len-between (min thing max)
(>= max (length thing) min))

(define-handler (source :is-stream? nil)
((room :string (len-between 0 room 16)))
(subscribe! (intern room :keyword) sock))

(define-handler (send-message)
((room :string (len-between 0 room 16))
(name :string (len-between 1 name 64))
(message :string (len-between 5 message 256)))
(publish! (intern room :keyword)
(encode-json-to-string
((:name . ,name) (:message . ,message)))))

(define-handler (index) ()
(with-html-output-to-string (s nil :prologue t :indent t)
(:html
(:head (:script
:type "text/javascript"
:src "/static/js/interface.js"))
(:body (:div :id "messages")
(:textarea :id "input")
(:button :id "send" "Send")))))

While we are still working with Lisp code, this interface is starting to look almost like a declarative language, in which we state what we want our handlers to validate without thinking too much about how they are going to do it. What we are doing is building a domain-specific language (DSL) for handler functions; that is, we are creating a specific convention and syntax that allows us to concisely express exactly what we want our handlers to validate. This approach of building a small language to solve the problem at hand is frequently used by Lisp programmers, and it is a useful technique that can be applied in other programming languages.

### A DSL for Handlers

Now that we have a loose specification for how we want our handler DSL to look, how do we implement it? That is, what specifically do we expect to happen when we call define-handler? Let's consider the definition for send-message from above:

(define-handler (send-message)
((room :string (len-between 0 room 16))
(name :string (len-between 1 name 64))
(message :string (len-between 5 message 256)))
(publish! (intern room :keyword)
(encode-json-to-string
((:name . ,name) (:message . ,message)))))

What we would like define-handler to do here is:

1. Bind the action (publish! ...) to the URI /send-message in the handlers table.
2. When a request to this URI is made:
• Ensure that the HTTP parameters room, name and message were included.
• Validate that room is a string no longer than 16 characters, name is a string of between 1 and 64 characters (inclusive) and that message is a string of between 5 and 256 characters (also inclusive).
3. After the response has been returned, close the channel.

While we could write Lisp functions to do all of these things, and then manually assemble the pieces ourselves, a more common approach is to use a Lisp facility called macros to generate the Lisp code for us. This allows us to concisely express what we want our DSL to do, without having to maintain a lot of code to do it. You can think of a macro as an "executable template" that will be expanded into Lisp code at runtime.

Here's our define-handler macro9:

(defmacro define-handler
((name &key (is-stream? t) (content-type "text/html")) (&rest args)
&body body)
(if is-stream?
(bind-handler
,name (make-closing-handler
(:content-type ,content-type)
,args ,@body))
(bind-handler
,name (make-stream-handler ,args ,@body))))

It delegates to three other macros (bind-handler, make-closing-handler, make-stream-handler) that we will define later. make-closing-handler will create a handler for a full HTTP request/response cycle; make-stream-handler will instead handle an SSE message. The predicate is-stream? distinguishes between these cases for us. The backtick and comma are macro-specific operators that we can use to "cut holes" in our code that will be filled out by values specified in our Lisp code when we actually use define-handler.

Notice how closely our macro conforms to our specification of what we wanted define-handler to do: If we were to write a series of Lisp functions to do all of these things, the intent of the code would be much more difficult to discern by inspection.

### Expanding a Handler

Let's step through the expansion for the send-message handler so that we better understand what is actually going on when Lisp "expands" our macro for us. We'll use the macro expansion feature from the SLIME Emacs mode to do this. Calling macro-expander on define-handler will expand our macro by one "level", leaving our helper macros in their still-condensed form:

(BIND-HANDLER
SEND-MESSAGE
(MAKE-CLOSING-HANDLER
(:CONTENT-TYPE "text/html")
((ROOM :STRING (LEN-BETWEEN 0 ROOM 16))
(NAME :STRING (LEN-BETWEEN 1 NAME 64))
(MESSAGE :STRING (LEN-BETWEEN 5 MESSAGE 256)))
(PUBLISH! (INTERN ROOM :KEYWORD)
(ENCODE-JSON-TO-STRING
((:NAME ,@NAME) (:MESSAGE ,@MESSAGE))))))

Our macro has already saved us a bit of typing by substituting our send-message specific code into our handler template. bind-handler is another macro which maps a URI to a handler function on our handlers table; since it's now at the root of our expansion, let's see how it is defined before expanding this further.

(defmacro bind-handler (name handler)
(assert (symbolp name) nil "name must be a symbol")
(let ((uri (if (eq name 'root) "/" (format nil "/~(~a~)" name))))
(progn
(when (gethash ,uri *handlers*)
(warn ,(format nil "Redefining handler '~a'" uri)))
(setf (gethash ,uri *handlers*) ,handler))))

The binding happens in the last line: (setf (gethash ,uri *handlers*) ,handler), which is what hash-table assignments look like in Common Lisp (modulo the commas, which are part of our macro). Note that the assert is outside of the quoted area, which means that it'll be run as soon as the macro is called rather than when its result is evaluated.

When we further expand our expansion of the send-message define-handler above, we get:

(PROGN
(WHEN (GETHASH "/send-message" *HANDLERS*)
(WARN "Redefining handler '/send-message'"))
(SETF (GETHASH "/send-message" *HANDLERS*)
(MAKE-CLOSING-HANDLER
(:CONTENT-TYPE "text/html")
((ROOM :STRING (LEN-BETWEEN 0 ROOM 16))
(NAME :STRING (LEN-BETWEEN 1 NAME 64))
(MESSAGE :STRING (LEN-BETWEEN 5 MESSAGE 256)))
(PUBLISH! (INTERN ROOM :KEYWORD)
(ENCODE-JSON-TO-STRING
((:NAME ,@NAME) (:MESSAGE ,@MESSAGE)))))))

This is starting to look more like a custom implementation of what we would have written to marshal a request from a URI to a handler function, had we written it all ourselves. But we didn't have to!

We still have make-closing-handler left to go in our expansion. Here is its definition:

(defmacro make-closing-handler
((&key (content-type "text/html")) (&rest args) &body body)
(lambda (sock parameters)
(declare (ignorable parameters))
,(arguments
args
(let ((res (make-instance
'response
:content-type ,content-type
:body (progn ,@body))))
(write! res sock)
(socket-close sock)))))

So making a closing-handler involves making a lambda, which is just what you call anonymous functions in Common Lisp. We also set up an interior scope that makes a response out of the body argument we're passing in, performs a write! to the requesting socket, then closes it. The remaining question is, what is arguments?

(defun arguments (args body)
(loop with res = body
for arg in args
do (match arg
((guard arg-sym (symbolp arg-sym))
(setf res (let ((,arg-sym ,(arg-exp arg-sym))) ,res)))
((list* arg-sym type restrictions)
(setf res
(let ((sym (or (type-expression
(arg-exp arg-sym)
type restrictions)
(arg-exp arg-sym))))
(let ((,arg-sym ,sym))
,@(awhen (type-assertion arg-sym type restrictions)
((assert-http ,it)))
,res)))))
finally (return res)))

Welcome to the hard part. arguments turns the validators we registered with our handler into a tree of parse attempts and assertions. type-expression, arg-exp, and type-assertion are used to implement and enforce a "type system" for the kinds of data we're expecting in our responses; we'll discuss them in HTTP "Types". Using this together with make-closing-handler would implement the validation rules we wrote here:

(define-handler (send-message)
((room :string (>= 16 (length room)))
(name :string (>= 64 (length name) 1))
(message :string (>= 256 (length message) 5)))
(publish! (intern room :keyword)
(encode-json-to-string
((:name . ,name) (:message . ,message)))))

...as an "unrolled" sequence of checks needed to validate the request:

(LAMBDA (SOCK #:COOKIE?1111 SESSION PARAMETERS)
(DECLARE (IGNORABLE SESSION PARAMETERS))
(LET ((ROOM (AIF (CDR (ASSOC :ROOM PARAMETERS))
(URI-DECODE IT)
(ERROR (MAKE-INSTANCE
'HTTP-ASSERTION-ERROR
:ASSERTION 'ROOM)))))
(ASSERT-HTTP (>= 16 (LENGTH ROOM)))
(LET ((NAME (AIF (CDR (ASSOC :NAME PARAMETERS))
(URI-DECODE IT)
(ERROR (MAKE-INSTANCE
'HTTP-ASSERTION-ERROR
:ASSERTION 'NAME)))))
(ASSERT-HTTP (>= 64 (LENGTH NAME) 1))
(LET ((MESSAGE (AIF (CDR (ASSOC :MESSAGE PARAMETERS))
(URI-DECODE IT)
(ERROR (MAKE-INSTANCE
'HTTP-ASSERTION-ERROR
:ASSERTION 'MESSAGE)))))
(ASSERT-HTTP (>= 256 (LENGTH MESSAGE) 5))
(LET ((RES (MAKE-INSTANCE
'RESPONSE :CONTENT-TYPE "text/html"
:COOKIE (UNLESS #:COOKIE?1111
(TOKEN SESSION))
:BODY (PROGN
(PUBLISH!
(INTERN ROOM :KEYWORD)
(ENCODE-JSON-TO-STRING
((:NAME ,@NAME)
(:MESSAGE ,@MESSAGE))))))))
(WRITE! RES SOCK)
(SOCKET-CLOSE SOCK))))))

This gets us the validation we need for full HTTP request/response cycles. What about our SSEs? make-stream-handler does the same basic thing as make-closing-handler, except that it writes an SSE rather than a RESPONSE, and it calls force-output instead of socket-close because we want to flush data over the connection without closing it:

(defmacro make-stream-handler ((&rest args) &body body)
(lambda (sock parameters)
(declare (ignorable parameters))
,(arguments
args
(let ((res (progn ,@body)))
(write! (make-instance
'response
:keep-alive? t
:content-type "text/event-stream")
sock)
(write!
(make-instance 'sse :data (or res "Listening..."))
sock)
(force-output
(socket-stream sock))))))

(defmacro assert-http (assertion)
(unless ,assertion
(error (make-instance
'http-assertion-error
:assertion ',assertion))))

assert-http is a macro that creates the boilerplate code we need in error cases. It expands into a check of the given assertion, throws an http-assertion-error if it fails, and packs the original assertion along in that event.

(defmacro assert-http (assertion)
(unless ,assertion
(error (make-instance
'http-assertion-error
:assertion ',assertion))))

### HTTP "Types"

In the previous section, we briefly touched on three expressions that we're using to implement our HTTP type validation system: arg-exp, type-expression and type-assertion. Once you understand those, there will be no magic left in our framework. We'll start with the easy one first.

#### arg-exp

arg-exp takes a symbol and creates an aif expression that checks for the presence of a parameter.

(defun arg-exp (arg-sym)
(aif (cdr (assoc ,(->keyword arg-sym) parameters))
(uri-decode it)
(error (make-instance
'http-assertion-error
:assertion ',arg-sym))))

Evaluating arg-exp on a symbol looks like:

HOUSE> (arg-exp 'room)
(AIF (CDR (ASSOC :ROOM PARAMETERS))
(URI-DECODE IT)
(ERROR (MAKE-INSTANCE
'HTTP-ASSERTION-ERROR
:ASSERTION 'ROOM)))
HOUSE>

We've been using forms like aif and awhen without understanding how they work, so let's take some time to explore them now.

Recall that Lisp code is itself represented as a tree. That's what the parentheses are for; they show us how leaves and branches fit together. If we step back to what we were doing in the previous section, make-closing-handler calls a function called arguments to generate part of the Lisp tree it's constructing, which in turn calls some tree-manipulating helper functions, including arg-exp, to generate its return value.

That is, we've built a small system that takes a Lisp expression as input, and produces a different Lisp expression as output. Possibly the simplest way of conceptualizing this is as a simple Common–Lisp-to-Common–Lisp compiler that is specialized to the problem at hand.

A widely used classification of such compilers is as anaphoric macros. This term comes from the linguistic concept of an anaphor, which is the use of one word as a substitute for a group of words that preceded it. aif and awhen are anaphoric macros, and they're the only ones that I tend to often use. There are many more availabile in the anaphora package.

As far as I know, anaphoric macros were first defined by Paul Graham in an OnLisp chapter. The use case he gives is a situation where you want to do some sort of expensive or semi-expensive check, then do something conditionally on the result. In the above context, we're using aif to do a check the result of an alist traversal.

(aif (cdr (assoc :room parameters))
(uri-decode it)
(error (make-instance
'http-assertion-error
:assertion 'room)))

This takes the cdr of looking up the symbol :room in the association list parameters. If that returns a non-nil value, uri-decode it, otherwise throw an error of the type http-assertion-error.

In other words, the above is equivalent to:

(let ((it (cdr (assoc :room parameters))))
(if it
(uri-decode it)
(error (make-instance
'http-assertion-error
:assertion 'room))))

Strongly-typed functional languages like Haskell often use a Maybe type in this situation. In Common Lisp, we capture the symbol it in the expansion as the name for the result of the check.

Understanding this, we should be able to see that arg-exp is generating a specific, repetitive, piece of the code tree that we eventually want to evaluate. In this case, the piece that checks for the presence of the given parameter among the handlers' parameters. Now, let's move onto...

#### type-expression

(defgeneric type-expression (parameter type)
(:documentation
"A type-expression will tell the server
how to convert a parameter from a string to
a particular, necessary type."))
...
(defmethod type-expression (parameter type) nil)

This is a generic function that generates new tree structures (coincidentally Lisp code), rather than just a function. The only thing the above tells you is that by default, a type-expression is NIL. Which is to say, we don't have one. If we encounter a NIL, we use the raw output of arg-exp, but that doesn't tell us much about the most common case. To see that, let's take a look at a built-in (to :house) define-http-type expression.

(define-http-type (:integer)
:type-expression (parse-integer ,parameter :junk-allowed t)
:type-assertion (numberp ,parameter))

An :integer is something we're making from a parameter by using parse-integer. The junk-allowed parameter tells parse-integer that we're not confident the data we're giving it is actually parseable, so we need to make sure that the returned result is an integer. If it isn't, we get this behaviour:

HOUSE> (type-expression 'blah :integer)
(PARSE-INTEGER BLAH :JUNK-ALLOWED T)
HOUSE>

define-http-handler10 is one of the exported symbols for our framework. This lets our application programmers define their own types to simplify parsing above the handful of "builtins" that we give them (:string, :integer, :keyword, :json, :list-of-keyword and :list-of-integer).

(defmacro define-http-type ((type) &key type-expression type-assertion)
(with-gensyms (tp)
(let ((,tp ,type))
,@(when type-expression
((defmethod type-expression (parameter (type (eql ,tp)))
,type-expression)))
,@(when type-assertion
((defmethod type-assertion (parameter (type (eql ,tp)))
,type-assertion))))))

It works by creating type-expression and type-assertion method definitions for the type being defined. We could let users of our framework do this manually without much trouble; however, adding this extra level of indirection gives us, the framework programmers, the freedom to change how types are implemented without forcing our users to re-write their specifications. This isn't just an academic consideration; I've personally made radical changes to this part of the system when first building it, and was pleased to find that I had to make very few edits to the applications that depended on it.

Let's take a look at the expansion of that integer definition to see how it works in detail:

(LET ((#:TP1288 :INTEGER))
(DEFMETHOD TYPE-EXPRESSION (PARAMETER (TYPE (EQL #:TP1288)))
(PARSE-INTEGER ,PARAMETER :JUNK-ALLOWED T))
(DEFMETHOD TYPE-ASSERTION (PARAMETER (TYPE (EQL #:TP1288)))
(NUMBERP ,PARAMETER)))

As we said, it doesn't reduce code size by much, but it does prevent us from needing to care what the specific parameters of those methods are, or even that they're methods at all.

#### type-assertion

Now that we can define types, let's look at how we use type-assertion to validate that a parse satisfies our requirements. It, too, takes the form of a complementary defgeneric/defmethod pair just like type-expression:

(defgeneric type-assertion (parameter type)
(:documentation
"A lookup assertion is run on a parameter
immediately after conversion. Use it to restrict
the space of a particular parameter."))
...
(defmethod type-assertion (parameter type) nil)

Here's what this one outputs:

HOUSE> (type-assertion 'blah :integer)
(NUMBERP BLAH)
HOUSE>

There are cases where type-assertion won't need to do anything. For example, since HTTP parameters are given to us as strings, our :string type assertion has nothing to validate:

HOUSE> (type-assertion 'blah :string)
NIL
HOUSE>

### All Together Now

We did it! We built a web framework on top of an event-driven webserver implementation. Our framework (and handler DSL) defines new applications by:

• Mapping URLs to handlers;
• Defining handlers to enforce the type safety and validation rules on requests;
• Optionally specifying new types for handlers as required.

Now we can describe our application like this:

(defun len-between (min thing max)
(>= max (length thing) min))

(define-handler (source :is-stream? nil)
((room :string (len-between 0 room 16)))
(subscribe! (intern room :keyword) sock))

(define-handler (send-message)
((room :string (len-between 0 room 16))
(name :string (len-between 1 name 64))
(message :string (len-between 5 message 256)))
(publish! (intern room :keyword)
(encode-json-to-string
((:name . ,name) (:message . ,message)))))

(define-handler (index) ()
(with-html-output-to-string (s nil :prologue t :indent t)
(:html
(:head (:script
:type "text/javascript"
:src "/static/js/interface.js"))
(:body (:div :id "messages")
(:textarea :id "input")
(:button :id "send" "Send")))))

(start 4242)

Once we write interface.js to provide the client-side interactivity, this will start an HTTP chat server on port 4242 and listen for incoming connections.

1. One solution to this problem is to force the clients to poll the server. That is, each client would periodically send the server a request asking if anything has changed. This can work for simple applications, but in this chapter we're going to focus on the solutions available to you when this model stops working.

2. We could consider a more general system that handles $N$ concurrent users with $M$ threads for some configurable value of $M$; in this model, the $N$ connections are said to be multiplexed across the $M$ threads. In this chapter, we are going to focus on writing a program where $M$ is fixed at 1; however, the lessons learned here should be partially applicable to the more general model.

3. This nomenclature is a bit confusing, and has its origin in early operating-systems research. It refers to how communication is done between multiple concurrent processes. In a thread-based system, communication is done through a synchronized resource such as shared memory. In an event-based system, processes generally communicate through a queue where they post items that describe what they have done or what they want done, which is maintained by our single thread of execution. Since these items generally describe desired or past actions, they are referred to as 'events'.

4. See for another take on this problem.

5. Pronounced "kloss", "see-loss" or "see-lows", depending on who you talk to.

6. The Julia programming language takes a similar approach to object-oriented programming; you can learn more about it in .

7. We're incidentally introducing some new syntax here. This is our way of declaring a mutable variable. It has the form (defparameter <name> <value> <optional docstring>).

8. with-timeout has different implementations on different Lisps. In some environments, it may create another thread or process to monitor the one that invoked it. While we'd only be creating at most one of these at a time, it is a relatively heavyweight operation to be performing per-write. We might want to consider an alternative approach in those environments.

9. I should note, the below code-block is VERY unconventional indentation for Common Lisp. Arglists are typically not broken up over multiple lines, and are usually kept on the same line as the macro/function name. I had to do it to stick to the line-width guidelines for this book, but would otherwise prefer to have longer lines that break naturally at places dictated by the content of the code.

10. This macro is difficult to read because it tries hard to make its output human-readable, by expanding NILs away using ,@ where possible.


text:
# Simplification Aptitude Important Formulas, Definitions, & Examples:

#### Overview:

 Topic Included: Formulas, Definitions & Exmaples. Main Topic: Quantitative Aptitude. Quantitative Aptitude Sub-topic: Simplification Aptitude Notes & Questions. Questions for practice: 10 Questions & Answers with Solutions.

#### Simplification Sequence:

For simplifying a question you must follow the sequence. Remember the word "VBODMAS". $$1 \longrightarrow V \longrightarrow {Vinculum \ or \ Bar \ (\bar{N})}$$ $$2 \longrightarrow B \longrightarrow Brackets, \ (1), \left\{2\right\}, [3]$$ $$O \longrightarrow {of}$$ $$3 \longrightarrow D \longrightarrow {Division (\div)}$$ $$4 \longrightarrow M \longrightarrow {Multiplication (\times)}$$ $$5 \longrightarrow A \longrightarrow {Addition (+)}$$ $$6 \longrightarrow S \longrightarrow {Subtraction (-)}$$

Under the Vinculum or bar $$(\bar{N})$$, Here N is any number

Under the brackets, solve brackets as numbered inside the brackets $$(1), \left\{2\right\}, [3]$$.

Example(1): $$\left[(6)^3 \times (3)^2\right] \div (2)^3 = ?$$

Solution: $$\left[(6)^3 \times (3)^2\right] \div (2)^3$$ $$\left[216 \times 9\right] \div 8$$ $$1944 \div 8 = 243$$

Example(2): $$216 \div 36 + 4 - 2 = ?$$

Solution: $$216 \div 36 + 4 - 2$$ $$6 + 4 - 2$$ $$10 - 2 = 8$$

Example(3): $$\left[{(20)^3 \div (8 + 2)^2 \times 2}\right] + 12 - 8 = ?$$

Solution: $$\left[{(20)^3 \div (8 + 2)^2 \times 2}\right] + 12 - 8$$ $$\left[{8000 \div 100 \times 2}\right] + 12 - 8$$ $$\left[{80 \times 2}\right] + 12 - 8$$ $$\left[160\right] + 12 - 8$$ $$160 + 12 - 8$$ $$172 - 8 = 164$$

#### Important Algebric Formulae:

$$(a + b)^2 = a^2 + 2ab + b^2$$ $$(a - b)^2 = a^2 - 2ab + b^2$$ $$(a^2 + b^2) = (a - b)^2 + 2ab$$ $$(a^2 - b^2) = (a + b) \ (a - b)$$ $$(a + b)^3 = a^3 + b^3 + 3ab \ (a + b)$$ $$(a - b)^3 = a^3 - b^3 - 3ab \ (a - b)$$ $$(a^3 + b^3) = (a + b) \ (a^2 + b^2 - ab)$$ $$(a^3 - b^3) = (a - b) \ (a^2 + b^2 + ab)$$

format: plain


text:
# Where effort joint values come from

Dear all,

When I subscribe to the topic /joint_states I get the name of the joints, the angular positions of these joints and the current effort values. I assume that the effort values are in M.m, but my question is, where these effort values come from? The effort values should depend on the force and torque values of the parent and the child link of a joint. Torque and forces are denoted by 3D vectors (x,y,z), in N.m as well. So how can I get these effort values if I have the 3D force and torque vectors of the parent and child links?

Thank you!!!

edit retag close merge delete

It doesn't matter for the more fundamental question, but can you please give a bit of context? Are you referring to a specific /joint_states topic (ie: published by a specific driver)?

( 2018-10-29 03:12:29 -0500 )edit

Yes exactly. For example: rostopic echo /joint_states or for any specific group like rostopic echo /left_arm/joint_states.

Thank you

( 2018-10-30 22:49:20 -0500 )edit

left_arm? Are you using a baxter?

Please provide a bit more context.

( 2018-10-31 03:27:14 -0500 )edit

No, I am using a Kinova Jaco arm attached to a Kinova Movo base

( 2018-10-31 17:57:02 -0500 )edit

I also want to know this question! I use UR5, which has no torque sensor on each joint/link. So the values are from motor currents or a computed torques?

( 2021-04-28 03:50:10 -0500 )edit

ok, I got the result of UR5. /joint_states/effort is motor current, see this issue.

About the unit of effort, you could see this link. The effort is Nm for revolute.

( 2021-04-29 02:07:34 -0500 )edit

@Yummy_Ou: all messages are documented. For JointState: sensor_msgs/JointState:

# This is a message that holds data to describe the state of a set of torque controlled joints.
#
# The state of each joint (revolute or prismatic) is defined by:
#  * the position of the joint (rad or m),
#  * the velocity of the joint (rad/s or m/s) and
#  * the effort that is applied in the joint (Nm or N).

which all makes sense, as those are the appropriate (derived) SI units, which adheres to REP 103.

( 2021-04-29 02:26:28 -0500 )edit


text:
## Yes, 1+2+3+… = -1/12. Sort of.

January 26, 2014

[Update: fixed an error that, while embarrassing, has no real bearing on anything, except perhaps to reinforce the admonition to be careful…]

The internet has been doing a whole lot of hating on this Numberphile video demonstrating (or, if you’re one of the haters, pretending to demonstrate through rank chicanery) that $1 + 2 + 3 + 4 + ... = - \frac{1}{12}$. I first saw it via Phil Plait of Slate’s Bad Astronomy blog, who followed his initial mind-blown and gushing post with a rather abject retraction. Some bloggers seem to be taking it quite personally, perhaps outraged that the charlatan Numberphilics are sullying the august and ethereal beauty of mathametics, ruining it for the rest of us and deluding the ignorant masses. Or something. I don’t get it, maybe because the original video is, if awfully (and admittedly) loose and sloppy and glib, pretty much correct.

Much of the vitriol is in the form of complaints that the manipulations in the video are not valid, that most manipulations with infinite series are not in fact valid, that one has to be very careful with infinite series, and that it’s easy to prove anything you want with innocuous-looking sleight of hand. That’s true: it is very easy to get these things wrong. I agree that Numberphile could have been more emphatic in pointing this out. But everything they do is legit. It’s a near thing — what they do in one context does not work in another — but it’s not by accident or legerdemain that their demonstration gives the right answer.

The first thing to get out of the way is that no, of course the sum doesn’t add up to -1/12 in the conventional way that you learned in elementary school. No infinite series has a sum in quite the way that 1+2=3. When we talk about sums of inifinite series what we really mean is that we have a summation method, a procedure that takes a series and produces a number, the “sum,” and that the procedure has some set of nice properties. Cauchy summation, taking limits of partial sums, is a particularly nice summation method with particularly nice properties, so nice that we have a special name — “convergent” — for the series for which it works. But it’s worth remembering that taking limits is already a non-trivial and potentially subtle operation.

In practice, what it means to say that $1 + 2 + 3 + 4 + ... = - \frac{1}{12}$ is that if you find yourself adding up all the positive integers and expecting to get an answer, that answer is very likely to be $-\frac{1}{12}$. (Or that you’ve made a mistake somewhere, always a possibility.) That’s certainly the case in string theory, where one common treatment is to hit the infinite sum, point out that it’s hard, and then show from other constraints on the theory that the only workable answer is $-\frac{1}{12}$. Yes, we could and arguably should make up some other notation, but as far as I know there’s nothing standard, and we (especially physicists) abuse notation all the time. It is not nonsense, and it really is deep. And weird.

The usual way to justify the result, and the one that most of the internet thinks Numberphile should have used, is through the Riemann zeta function. In fact, Numberphile did use that in their follow-on video and in Tony Padilla’s response to the whole kerfuffle. That’s fine and correct and all, but (IMO) not a very intuitive way of looking at it. What do zeta functions and analytic continuations have to do with this? Lots, actually, but I’m dense and have always found it to be rather opaque. But there are other ways of looking at the problem, to my taste much more satisfying. One fortuitous consequence of this brouhaha has been a lot of links to a post of Terry Tao’s of a few years ago, which I have found enormously helpful. It’s pretty dense, especially if you’re not a mathematician, but worth reading even if you finding yourself glazing over at Bernoulli numbers. It is my chief inspiration in what follows (with the usual caveat the everything I have gotten wrong herein is my fault alone).

Beginning to hint at formality, let’s look at summation methods and their properties. Then we can prove things of the form “if this summation method has these properties, and it works for these series, then it must assign this series this value.” That’s one way to interpret the video sensibly. The problem is that different summation methods allow different manipulations, and figuring out which you can use is subtle. The arguments in the video (if you interpret it as I do) require three different methods, which get progressively more restrictive and finicky about what techniques you can legitimately use. What each of the video’s steps relies on does not work at the next step!

So let’s talk about summation methods and their properties.

Properties first, mostly stolen from wikipedia:

• Regularity: the method gives the ordinary sum for convergent series.
• Linearity: the methods behaves as you would expect when you add series together or multiply them by constants.
• Stability: You can offset a series and get the same answer : $a_1 + a_2 + a_3 + ... = 0 + a_1 + a_2 + ...$. Not all methods, including ones we’ll use, are stable in this sense!
•  [no name that I know of!] We can “expand” the series by sticking zeros before every term and get the same answer:
$a_0 + a_1 + a_2 + ... = 0 + a_1 + 0 + a_2 + 0 + a_3 + ...$
This is not the same as inserting zeros in arbitrary places, including after each term — we can (and will) have series for which $a_0 + a_1 + a_2 + ... \neq a_1 + 0 + a_2 + 0 + a_3 + ...$ !

And some summation methods

• Cesàro summation: even if the partial sums don’t converge, their averages might, and if they do call that the sum.
• Abel summation: if $a_0 + a_1 x + a_2 x^2 + a_3 x^3 + ...$ converges for $x<1$, and if $\lim_{x \to 1} \sum a_i x^i$ exists, call that the sum.
• “Polynomial renormalization”: I don’t know what to call it, but what Terry Tao does in his deservedly much-quoted blog post mentioned above. Basically, we first “regularize” the sum by applying a smooth cutoff function that is 1 at 0 and 0 for $x>=N$. If the result is a polynomial in N plus some residual term that goes to 0 as N goes to infinity, we use just the constant part of the polynomial as the sum. Not coincidentally, this is very reminiscent to what physicists often do in quantum field theory.

Cesàro summation and Abel summation have all the properties I listed; polynomial renormalization, however, is not stable. Any series that is Cesàro summable is also Abel summable (and both give the same answer). I think that Abel summability implies and is consistent with polynomial renormalization (there is probably a proof lurking in the appropriate section of Tao’s article, but I haven’t figured out whether it works here). The series here, however, are all PR-summable, consistently with Abel summation for the ones that are Abel summable.

$S_1 = 1 - 1 + 1 - 1 + ...$.

This is obviously divergent — the partial sums oscillate between 1 and 0 — but it Cesàro summable, as the partials averge out to $\frac{1}{2}$.
Next we look at

$S_2 = 1 - 2 + 3 - 4 + ...$.

This is Abel summable, and we could easily use the explicit formula to calculate its value, but instead we’ll just assume it and calculate indirectly, using linearity and stability and the value we already have for Grandi’s series:

$\begin{array}{ll} & 1 - 2 + 3 - 4 + ... \\ + & 0 + 1 - 2 + 3 - ... \\ = & 1 - 1 + 1 - 1 + ... \\ = & \frac{1}{2} \end{array}$

so $S_2 + S_2 = \frac{1}{2}$ and $S_2 = \frac{1}{4}$.

Now things get subtle. Let’s assume that this carries through to PR summability (it does), and continue. We can no longer use stability — no more offsetting a series — but we can use the expansion-by-zeros trick if we do it right. We compute

$\begin{array}{llc} & &1 + 2 + 3 + 4 + ... \\ - & 4 \times & ( 0 + 1 + 0 + 2 + ... ) \\ = & & 1 - 2 + 3 - 4 + ... \end{array}$

from which we find $S - 4S = \frac{1}{4}$, or $S = -\frac{1}{12}$.

To recap, we

• used Cesàro summation to compute a “sum” for Grandi’s series
• assumed that $S_1$ is Abel summable, and computed its value using Grandi’s series and the properties of Abel summation, and finally
• used that value and the properties of PR summation to compute the original sum.

As promised steps two and three were of the form “if this method works, the result has to be that.” It turns out that they do!

It’s worth looking at some manipulations that superficially appear to be every bit as valid as what we’ve just done but don’t work. First, I’ll state without proof that PR summation leads to the result that

$1 + 1 + 1 + ... = -\frac{1}{2}$

If we subtract this from $1 + 2 + 3 + ...$, we get

$0 + 1 + 2 + 3 + ... = \frac{5}{12} \neq -\frac{1}{12}$

— I warned you offset wouldn’t work here. I also warned you that you couldn’t “expand” a series with zeros in the wrong place, and in fact we can calculate that

$1 + 0 + 2 + 0 + 3 + ... = \frac{1}{24}$

Go figure! Some others:

$0 + 1 + 0 + 1 + ... = -\frac{1}{2} \\ 1 + 0 + 1 + 0 + ... = 0 \\ 0 + 1 + 1 + 1 + ... = -\frac{3}{2}$

So it behooves you to be careful.

## Cat Sonnet

November 9, 2013

Hugo waits, his every muscle taut,
Except for twitching tail uncanny still.
Soon, suddenly, he’ll strike, as swift as thought:
Soon hapless mouse will be but bloody kill.
Then later, after napping, he may play —
Or is it fight? there’s no good way to tell.
Perhaps his human servants’ flesh he’ll flay,
Or make his cat companion’s life a hell.

And now he craves attention, tame and meek,
Imploring only to be stroked and scratched.
He purrs and stretches, lang’rous, long, and sleek.
Gentleness and innocence are matched
With casual violence in the feline race,
Dual natures joined in feline grace.

## The Thin Place

October 31, 2013

The borders of the world are pale now,
See how they grow very thin.
Near to us loom other places
Home to things not quite our kin.
What doors to otherwhere may open,
And what may come in?

An impish goblin, bent on mischief?
A Faerie prince from Tír na nóg?
A witch with her familiar daemon,
Ghostly cat or spectral dog?
Baleful and mysterious figures
Creeping through the fog?

Souls that should be long departed;
Wights and wraiths with sorrow crowned;
Boggarts, brownies, keening banshees
Sounding their unearthly sound;
Kobolds, dwarves, and little men
From caverns underground.

Do not depart, nor take your leave!
Pull the blinds and bar the door ‘gainst
Things of which you daren’t conceive!
Unless you want to see what visits
On All Hallows’ Eve.

## Kindle Fire

November 17, 2011

For the first time in my life, I seem to be the first person I’ve talked to to have the latest electronic toy, one of these. It doesn’t have the enormous coolness factor of something shiny and made by Apple, but still.

Overall I’m loving it. I’m using it more as an extra-portable and convenient web-browser/time-waster than as a reader and Amazon-content-delivery system (its obvious purpose), more cheap-and-small iPad than expensive Kindle. Which was pretty much my plan. It works great for that. It even works for some things I honestly didn’t expect it to, like reading powerpoint presentations (I wasn’t surprised so much that it tried as that it succeeded).

Of course there are some annoying things about it. Many would (I assume) apply to any mobile touch-screen toy—typing is painful, mobile websites are missing bits I take for granted (e.g. formatting in gmail). Some (many? most?) android apps don’t seem to be available in the amazon appstore—that is probably changing even as I write.

But the single most annoying thing is something I had expected to be trivial. I had figured it would be trivial to download a free ebook—from Project Gutenberg, say—and start reading it immediately. Not so! You can download a .mobi file, and then it just sits there refusing to be read. The quickest way I’ve found to get the demmed things to import is to

• open QuickOffice
• Hold your finger down on the file, and pick “Send” when the menu comes up
• Email it to your kindle account
• Wait a while. Wait some more.

Isn’t there a better way to do this? If not, for the love of heaven why?*

On the subject of free ebooks, I’d like to give a shoutout to someone calling herself Cthulhuchick for digitizing the complete works of H. P. Lovecraft and making them available in a nice free edition. Also for picking a great nom d’internet.

* Maybe I’ll answer my own question: Amazon has no interest in my reading stuff I didn’t buy from them.

## Out of Bembo

November 15, 2011

I just got a copy of Out of Oz from the library, and the first thing I notice is that the typeface is back to the Truesdell of the original Wicked, rather than the always lovely but comparatively bourgeois and anodyne Bembo of Son of a Witch and A Lion Among Men. I hope this means the book is as weird as the first one was.

## In Other News, Sun to Rise in East

November 15, 2011

In June 2010, Daniel Klein published a gratuitously silly piece in the Wall Street Journal’s gratuitously silly op-ed section, purporting to show that liberals are stupider than conservatives and libertarians, at least in terms of basic economics. This occasioned some amount of delight among conservatives and libertarians—vide Veronique de Rugy in the reliably obnoxious National Review Online (to be fair, not all conservative pundits fell for it). The “methodology” Klein relied on centered around a series of agree-or-disagree statements, things like “Third World workers working for American companies overseas are being exploited” and “Minimum wage laws raise unemployment,” most of which I would call at best “arguable,” all of which are pretty clearly politically charged in that they challenged various liberal positions, and all of whose “correct” answers were on the conservative/liberal sides of various issues. So, surprise! liberals did worse on the “do I agree with the conservative/liberaltarian author” test than conservatives and liberaltarian.1

To his (relative) credit, Klein eventually realized the problem, and he did a similar exercise with questions slanted the other way (“gun-control laws fail to reduce people’s access to guns”). To what should be no one’s surprise, this time he found that conservatives were the stupid ones. Read about it in the current Atlantic. Klein now thinks what both surveys demonstrate is “myside bias.” Really!

I feel a little sorry for Zeljka Buturovic, Klein’s (possibly more academic) collaborator. According to Klein’s Atlantic article, her goal wasn’t to prove anything about economic literacy, but rather to “[explore] the possibility that ideological differences stem more from differences in people’s beliefs about how the world works than from differences in their basic values.” There might be something to that, although I don’t know how to sort out the causality there (and I haven’t read the journal articles). Since she wasn’t the one who wrote either the WSJ or Atlantic articles, I don’t want to tar her with the same brush. Sorry, Zeljka!

Oh, another annoying thing. The questions and “correct” answers are reported in the most confusing way imaginable. From the Atlantic, citing the questions with the canonically incorrect answers in quotes:

Here’s what we came up with, again with the incorrect response in parentheses: a dollar means more to a poor person than it does to a rich person (disagree); making abortion illegal would increase the number of black-market abortions (disagree); legalizing drugs would give more wealth and power to street gangs and organized crime (agree); drug prohibition fails to reduce people’s access to drugs (agree); gun-control laws fail to reduce people’s access to guns (agree); by participating in the marketplace in the United States, immigrants reduce the economic well-being of American citizens (agree); …

So how many negatives to I have to wade through to figure out what an enlightened person thinks of gun control laws? I’m pretty sure Klein thinks they reduce access to guns (I think that too), but that’s more because that’s what you would ask if your goal is to prove conservatives are wrong than from trying to cancel out the “fail to” and the “incorrect response in parentheses.”

1. For what it’s worth, on the original set of questions I would probably have mostly given the “conservative” answers, but only if “it depends” wasn’t an option, and only if I had inferred the context that would be generally implied from the fact that someone was asking about that particular set of things and using that particular phrasing.

## node.js

November 13, 2011

I’ve been looking at node.js, the server-side (more general than that, really) javascript execution environment. My gut reaction is that I like it–I like it a lot. But I think it will take some serious investigation to determine whether it’s ready for an industrial-strength application.

Now some details, aimed at node-n00bs such as myself. Anyone who would like to point out how badly I’ve gotten things wrong, feel free!

First, it’s important to know that node.js is architected to solve a specific problem, to wit, scalability in the presence of lots of concurrent access. The usual way something like apache/php handles concurrency is to spawn a thread for each server request. But threads have overhead, and there’s only so far you can push that before you have to buy more servers.

node.js has pretty much the opposite philosophy. The buzzwords you see are “asynchronous” and “event-driven” or just “evented”—its central element is a single-threaded event loop. But that doesn’t tell you much about why it’s a good idea. I found a much more revealing tagline here: “everything runs in parallel, except your code.”

The idea is that in a typical (non-trivial) server request most of the processing time is taken up in things like database or filesystem access, henceforth referred to generically (and not always completely correctly) as “IO.” Those are things that either don’t take a lot of CPU cycles, or at least are already in their own threads or processes. If the thread running those IO operations waits for them to complete, it will be sitting idle; in a single-threaded event loop that means it will block anything waiting for it. So in a single-threaded event loop, you don’t wait! IO operations in node.js don’t return their data directly; instead, they accept callbacks to process the results when they’re ready. Those callbacks are themselves processed in the event loop.

The callback functions themselves should all be lightweight, so that tens of thousands of them can be executed per second. They palm off all the hard work to “IO functions,” black boxes that may in turn add more callbacks. The node.js API, and good node.js plugin modules, are structured to make it positively difficult to do anything that blocks.

Here’s a “hello world” webserver, straight from the node.js front page, that responds to any request with, er, “hello world”:

var http = require('http');
http.createServer(function (req, res) {
res.end('Hello World\n');
}).listen(1337, "127.0.0.1");

The guts of that there–the argument to http.createServer–is the callback that gets called from the event loop whenever the server fires the “request” event (you don’t see the event loop yourself; you just add callbacks for events, and they are called from the event loop). It does the actual responding by calling methods on res, a ServerResponse object.

Of course a real response will be more complicated (starting with url-parsing, which I’ll ignore completely). Traditionally that might look something like

http.createServer(function (req, res) {
var data = do_some_io_operation(req);
var output = do_some_more_processing(data);
res.end(output);
}).listen(1337, "127.0.0.1");

But those two function calls, if they do anything IO-ish or otherwise nontrivial, are not kosher node.js. This should be something like this:

http.createServer(function (req, res) {
do_some_io_operation(req, function(data) {
do_some_more_processing(data, function(output) {
res.end(output);
});
});
}).listen(1337, "127.0.0.1");

What happens on a request is:

• The handler calls do_some_io_operation, which registers a callback and fires off whatever the operation is. It–and the handler–return immediately, and processing can move on to anything that’s waiting in the event queue.
• When the io operation completes, the first inner callback gets executed. That calls do_some_more_processing, which registers yet another callback, starts whatever it starts, and returns.
• When THAT finishes, the inner callback finally takes all the data it now has available and finishes responding to the request.

Database access might look something like this (not real code, but it might be close, modulo error handling)

http.createServer(function (req, res) {
dbase.connect('tcp://whateverdbase@localhost/whatever', function(connection) {
var query = connection.query('SELECT * FROM some_table');
query.on('row', function(row) {
res.write('<p>' + row + '</p>');
});
query.on('end', function(row) {
res.end('Hello World\n');
});
});
}).listen(1337, "127.0.0.1");

Connecting to the dbase is an IO operation, and since everything depends on that almost the whole response function is inside a callback. Setting up the query as I have it here doesn’t necessarily do anything immediately, so it does NOT need to take a callback, but that would be an alternative API. However, you do have to wait for the results of the query, so it gets callbacks for its “row” and “end” events.

I rather like this functional quasi-continuation-passing style of programming, mostly because I like finding out the abstruse theoretical concepts turn out to be useful for real. It might get a bit messy in real examples, though. At the very least I wonder if it would call for a different indentation style .

[And I have a “solution” for that. Or at least something that’s kept me happily occupied for the last couple of days. More on it later, if I ever get around to it.]

The fact that this is all in javascript has a few real advantages. Javascript is halfway to being a functional language, and is thus well suited to this style of programming. But it’s not Lisp or Haskell, so existing programmers don’t have to rewire their brains to use it. (I love Haskell, but I can’t imagine trying to find and manage a team to write a Real Product with it.) Indeed, any web programmer will already be fluent in javascript, and used to working with callbacks, if not quite to the pervasive level that node.js requires.

Using the same language on the client and server is a nice benefit, too. It makes it easy to share code between the two sides, something that can be useful (caveat: writing javascript that will work properly in a browser and inside node.js is NOT completely trivial, but it’s usually not that difficult either). And it’s nice for us programmers to avoid the annoying context switches between languages. Going from javascript to python, for example, I am forever forgetting to put quotes around dictionary keys.

And compared to other dynamic languages, javascript on google’s V8 engine, which node uses, is really fast. For “pure” stuff, just function calls and for loops and the like, it appears to be more in the C/C++ range than the Python/Ruby/php range.

Now for the downsides. Actually, for all I know there aren’t any prohibitive ones! But node.js is still relatively new, and I although I think the core is relatively stable the general ecosystem isn’t. There are lots and lots of modules for doing various things, but in this sort of open-source world it’s really difficult to know what you can trust. In the Perl world, for example, I’ve seen CPAN hailed as the greatest thing since sliced bread, but I’ve seen a lot of crap there, and sorting through it can be a real cost.

node.js itself is quite low-level, lower-level even than php. Someone needs to encapsulate even simple things like (e.g.) gathering get and especially post data into dictionaries of query values and data (not that that’s hard, but it does need to happen). There are some embryonic higher-level frameworks—Express looks very promising, and at least does that post-data processing—I am fairly certain there’s nothing anywhere near as mature and trustworthy as Ruby on Rails or Django (or asp.net, if you like that sort of thing).

And conceptually not everyone agrees that eventing is the way to handle concurrency. There are lots of partisans of Erlang and the aforementioned Haskell and even of traditional threading who beg to differ. I’m a bit out of my depth here, so can’t comment usefully.

## Blame Ben Nelson!

August 3, 2011

In any sort of catastrophe it’s important to decide who to blame. A good scapegoat almost never helps to avoid future catastrophes, but who cares? Blame is fun!

So who do we blame for the Sugar-coated Satan Sandwich? Which on its own merits isn’t as bad as it could have been, really… at least civilization didn’t collapse, as was a real possibility; and the creation of the terrifyingly-named Super Congress has the great virtue of postponing the really awful decisions for a few months. No, the real problems with it are the precedents it set, and the fact that our governing class’s (including the punditocracy) has decided to respond to a genuine economic crisis by focusing on the single least relevant factor and as a result implementing the stupidest possible policies. Who can we blame for that?

The obvious answer for liberalish coastal elites like me is the Tea Party. But that’s too easy. Sure, the Tea Partiers are stupid and crazy, and it’s absolutely terrifying how much power they wield. But that is their nature, and you can’t expect them to go against their nature. They’ve said all along that they don’t like government, so by golly they’re going to try to get rid of it any way they can. They’re the mad bomber from Source Code (name whited out in case you don’t want to know what I’m very mildly spoiling) (paraphrasing, because I forget the real quote): “We can build a new and better society out of the rubble. We just need to make the rubble first.” No wonder they’re cross that they got everything they asked for: they didn’t hold the government and economy hostage to have their demands met; they made their demands so they could shoot the hostage. This is crazy, but, well, they’re crazy, and have a pretty credible insanity defense. “Forgive them, father, for they know not what they do.”

The sane Republicans, insofar as that’s not an oxymoron now, are considerably more culpable. They at least knew what they were doing, and I can only hope they’re full of self-loathing now. This goes double, of course, for Boehner and Cantor, who remind me of Count Baltar from the old Battlestar Galactica (Baltar from the NEW Battlestar Galactica is much more interesting), allying with the sworn enemies of their very existence. But I can almost understand them; like so many politicians through history, they’re acting out of fear, terrified for their jobs. Well, Cantor is trying to get a better job, but close. Again, you can’t expect someone to against their nature.

How about the Republicans’ corporate masters? We’re getting closer here. At the risk of having Godwin’s Law justifiably invoked, they are treading perilously close to the Weimar-era German business interests who backed the Nazis, thinking they would be easily-controlled tools. I do hope the Chamber of Commerce is reconsidering its priorities. But business leaders in our society, even the allegedly brilliant ones, are always short-sighted, greedy, and stupid, so again you sort of have to expect this.

Pundits? Journalists? Bloggers? Well, yes, they’re mostly pretty bad. But there’s only so much pundits can do to influence the national debate, as much as they would like to think otherwise.

It’s fashionable among my progressive brethren and sisteren to blame Obama for all this. Now I won’t pretend that I’m not disappointed in Obama, but honestly, what could he have done? Obama really wants to make deals and compromise and bring people together, not productive strategies in dealing with modern Republicans, but I honestly don’t know what anyone could have done, no matter how tough. You can’t play chicken with someone who WANTS a head-on collision. Yeah, I know, he should have seen this coming, but IIRC that was just after the LAST hostage crisis; even if Obama could have imagined the teabaggers were that crazy it would have been difficult to continue. It’s kind of like the way we didn’t attack the Russkies in August 1945.

So who do I blame? How did we get to this point anyway? The proximate cause is of course the great Shellacking of 2010, an inevitable result of a lousy economy and a high unemployment rate (and a fickle Democratic base). But one lousy election—one where only one house of Congress changed parties—really shouldn’t have sent the nation’s political discourse so quickly off the sanity cliff. I can easily imagine a parallel universe in which the Democrats lost the House not to the Tea Partiers but to more or less sane Republicans. My theory is that the underlying problem is in what should have been Obama’s greatest triumph, health care reform. You’d think that people would be pleased with some guarantee of health insurance, but no, they HATE it. And it was that hatred (say I) more than anything else that allowed the teahadists to sweep into office, so cowing Democrats that in their confusion and terror they willingly stipulated to the Republican view of the world.

Obviously much of the anti-Obamacare bile is due to Republican lies, not just the flashy ones like Death Panels, but the underlying lie that our health care system is The Best In The World (something I think you can only believe if you’ve never actually meant someone from another country). But lies can only get you so far. As I see it the real problem was with the Democrats themselves, specifically with the Senate Democrats, for doing such a miserable job of getting the the thing passed, and inviting the world in for an open house at the Sausage Factory. The most egregious example (that I know of; perhaps were even worse things that weren’t so obvious) was the legendarily odious Cornhusker Kickback—even though it didn’t make it into the final bill, it was so appalling that it rather justifiably convinced the world that the whole process was corrupt and its result inevitably evil. So, on behalf of the Tea Party, thank you Ben Nelson!

## Historical Writing

August 2, 2011

One more note on the previous post:

It’s not fair of me to expect the same level of juicy detail from ancient history as there is (or can be) in classical or modern history; there is simply more information about the more recent periods. As far as I know the Egyptians had no analogs of the wonderfully chatty and gossipy (and unreliable) classical historians, and although they were great record-keepers there just isn’t the same sort of detail we have about more recent times. But I’ll compare anyway. Here’s the sort of thing I love, from Strange Victory , by Ernest May (which I recommend!):

Canaris was a strange character. There were many such, of course, at the center of the Third Reich, but Canaris stands out among them…. Five feet four, with prematurely white hair, he detested tall men and, above all, tall men with small ears. The loves of his life were Seppel and Sabine, two dachshunds from whom he was almost inseparable. When traveling, he would require from his staff frequent reports on their apparent emotions and their bowel movements…

…[A]other possibility is anger over encroachments in his domain by Reinhard Heydrich, the head of Himmler’s security service. Outwardly, his relationship with Heydrich was cordial; the Canarises and the Heydrichs were neighbors and dined together. But Heydrich was thought repulsive by men who had no such reaction to Himmler or Bormann. Also, he was very tall and had small ears.

## The Rise and Fall of Ancient Egypt

August 2, 2011

For years I’ve loved ancient history without really knowing much about it. I’ve never been able to keep the Babylonians and Assyrians completely straight, let alone the Akkadians, and what any of them have to do with the Sumerians (my knowledge of whom comes primarily from having seen The Mole People at an impressionable age). I’ve never been quite sure of who the Hittites and Amorites and Mittanians and Chaldeans were. Those Sumerian city states have great names like Uruk and Ur and Lagash and Eridu but (apart from Abraham’s alleged origin in Ur of the Chaldees; doubtless he left to escape the mole people) I have no idea which is which or why we should care. The comings and goings of the MInoans and Mycenaeans and Achaeans and Dorians and other proto-Greeks are a mystery to me, providing only a little background for the Iliad and Odyssey and some half-remembered Mary Renault novels.

And then there’s Egypt. Egypt is far more familiar to us American rubes than Babylonians and Assyrians and Mittanians. It’s hard not to be exposed to all sorts of things about Egypt, romantic things like pyramids and mummies and sarcophagi for cats and animal-headed gods and King Tut. Yul Brynner and Anne Baxter (“Mo-o-o-ossses…”) are deeply ingrained as my Platonic forms of Pharaoh and Pharaohess (Pharaoness? Pharessa?). But the ancient Egyptian civilization lasted for 3,000 years. That is a long time, half as long again as it’s been since it it became part of Rome until now, and apart from “Cleopatra was at the end” all that history is compressed in our collective unconscious into an undifferentiated mass of pyramids and mummies and sarcophagi for cats and animal-headed gods and King Tut and Yul Brynner and Anne Baxter.

So when I read about Toby Wilkinson’s The Rise and Fall of Ancient Egypt, a popular history of the entire civilization, I hied me to the library and found a copy. It serves its purpose well: I’m much clearer—well, somewhat less vague—on what the difference between the Old and Middle and New Kingdoms, and where Memphis was (and why), and who the Hyksos and the Sea People were, and even to a certain extent which king was which and which did what. I know now where the word “Pharaoh” comes from (“Per-aa,” or “Great House,” applied metonymically to its inhabitant) and why (awkwardness about what to call Hatshepsut—there was no such thing as a “Queen”). The book is easy to read, and it zips right along. If you’re looking for a summary of all of ancient Egyptian history, bearing in mind that a “summary of all of ancient Egyptian history” will necessarily be a few hundred pages, this is it.

One of the highlights (I thought) comes at the very beginning of the Egyptian kingdom. Among the artifacts of that era, now a highlight of the Egyptian Museum of Cairo (and of Adrian Veidt’s—Ozymandias’s!—office in Watchmen) is the Narmer Palette. That’s “palette” as in a thing for mixing paints, in particular cosmetics; the ur-Eqyptians apparently used them as ceremonial objects in those days, a development of sacred face painting among in nomadic cultures or some such. The Narmer palette (apparently) commemorates the very unification of the Two Kingdoms of Egypt under Narmer, Egypt’s possibly mythical first king and founder of its First Dynasty. The obverse is full of Mespotamian motifs: the king as a bull destroying a walled town, those “serpopards” whose intertwined necks frame the pigment-mixing depression. The reverse is more distinctly Egyptian, with the now-human king wearing the crown of Upper Egypt. Although to my untrained and untrustworthy eye there is still some of Mesopotamia there—the king’s calf muscles look distinctly Assyrian to me, and I wonder how much those cows at the top (proto-Hathors, apparently) owe to Mesopotamian predecessors—the Narmer palette represents the very beginning not only of the Egyptian kingdom but of Egyptian, which (again to my untrained eye) remained remarkably consistent for millennia thereafter.

But this isn’t a perfect book. Reading it is a great first step in understanding Egyptian history, but all those kings still blur together. To some extent, I suppose that’s unavoidable—there were a lot of them, after all, and really I don’t know that there is always much to distinguish them. But Wilkinson’s (I hate to say it) somewhat cliched writing doesn’t really help. There are too many “brilliant demonstrations of the unite-and-rule concept” and “brilliant flashes of inspiration” and “brilliant but simple expedients.” I’m sure there is precious little that can really be said about any individual, king or otherwise, especially given Egyptian kings’ perennial habit of erasing all record of their predecessors, but I would desperately like more indivuating details, and anecdotes.

And I’d also like to know more about the Egyptian religion (or should that be religions?). Gods and priests and temples obviously permeate Egyptian history and life, but really all I can say I got from the book is that there were a confusingly large number of gods, whose cults were more or less important in various cities. How did those cults develop, and split and merge, and relate to each other in general? How did perceptions of the gods and details of their cults change over time? What did all those priests do all day, anyway?

So, definite thumbs up, but I am left wanting more. Which is certainly better than left wanting less—


text:
# Include vs Input in my case

Suppose you have two fully finished thesis, with their own bibliography and their own appendices. When I include one into the other, there is (obviously) some interference between the two different bibliographies and two different appendices.

So my question is: what is the optimal solution so that I get only one bibliography and one appendix. Also, I would like the included file to consist of one large section with subsections and not as it is at the moment, which is: each sections of the included file gets a section of its own.

• You might need to give more information about the structure of your documents, a latex document can only have one `\documentclass` so you can't include or input one in to another. If each document consists of `\including` a list of separate section files, you can of course make a combined document but including the combined set of sections, but without knowing the structure it is not possible to say much. – David Carlisle Sep 18 '13 at 15:53
• Voted to reopen. Not a duplicate of tex.stackexchange.com/questions/246/…, in spite of the misleading title, that should be something like How to combine two complete theses in a document?, since neither `\input` nor `\include` solve the real question. – Fran Sep 19 '13 at 2:55
As David Carlisle said, it is not possible say much with the actual information, but I suspect that you are looking for the `combine` class and the packages `combinet`, `combnat` and `combcite`.
With `texdoc combine` you will obtain all the information of what you need at this respect.
If this is not exactly what you want, take a look also to the packages `docmute` and `standalone` (the package, not the document class!).


text:
# Which proton in cyclohexene is the most acidic?

I know that $\mathrm{H^b}$ is most acidic due to conjugation (resonance).

But I am confused as to how to compare $\mathrm{H^a}$ and $\mathrm{H^c}$. I think $\mathrm{H^a}$ should be more acidic than $\mathrm{H^c}$ due to hyperconjugation.

• Have you been taught the relative acidity of alkanes, alkenes and alkynes? chemistry.stackexchange.com/questions/32341/… – orthocresol Jan 9 '16 at 12:38
• @orthocresol so the correct decreasing order should be $\mathrm{H^b}$ , $\mathrm{H^c}$ and $\mathrm{H^a}$ – user101522 Jan 9 '16 at 15:53
• Yes, according to this link the allylic proton (b) has a pKa ~36, the vinylic proton (c) ~43, and the sp3 proton (a) ~50. – orthocresol Jan 9 '16 at 16:25

When comparing acidities of protons, we need to find out what the corresponding conjugate bases are.

The conjugate base upon removing $\ce{H_\mathrm{b}}$ is an allyl anion. There is (and should be) no question that it is the most stable carbanion of the three we are going to consider.

The conjugate base upon removing $\ce{H_\mathrm{a}}$ is an $\mathrm{sp^3}$ configured carbanion while the conjugate base upon removing $\ce{H_\mathrm{c}}$ is an $\mathrm{sp^2}$ configured carbanion. This is nice, because it allows us to directly compare the two using their hybridisation.

In general, lone pairs (and that explicitly includes those coming from anions) want to be in the lowest-lying atomic or molecular orbital possible. Since an s orbital is lower in energy than a p orbital from the same shell, lone pairs prefer s orbitals. If there is no pure s orbital available — as would be the case in practically all carbon-centred lone pairs — an orbital with the highest possible s character is preferred instead. If we look at the configuration $\mathrm{sp^3}$ we can say that it has $25~\%$ s character while $\mathrm{sp^2}$ has $33~\%$ s character. Since $33~\% > 25~\%$, the $\mathrm{sp^2}$ configured carbanion is more stable — which is also supported by corresponding $\mathrm{p}K_\mathrm{a}$ values.

The method outlined above is a very crude approximation building on a high amount of simplifications but gives us the correct result in practically all carbon-centred anion cases.

Thus, the acidity order is: $$\ce{H_\mathrm{b}} > \ce{H_\mathrm{c}} > \ce{H_\mathrm{a}}$$

Note: Ron noted in a comment above that the acidity of an $\mathrm{sp^2}$ hydrogen almost parallels that of an allylic hydrogen. So while most courses may teach this simple ordering, you should keep that in mind as additional information.

To address the question of hyperconjugation, ${H^b}$ presents the classic example of hyperconjugation between the ${\sigma}$ bond and ${\pi^*}$. No hyperconjugation is possible between ${\pi^*}$ and the ${sp^2}$ orbital attached to ${H^c}$. However, some very weak interaction between ${\sigma}$ bond between $\ce{CH^a}$ and ${\pi^*}$ could lead to some weakening of the sigma bond and enhanced acidity.

If we were to remove $\ce{H^c}$, we would get a structure similar to a vinyl cation. The vinyl cation is highly unstable and hence it would be more difficult to remove $\ce{H^c}$,i.e. $\ce{H^c}$ is less acidic than $\ce{H^a}$. Also hyperconjugation is greater for $\ce{H^a}$ than it is for $\ce{H^c}$.

• Welcome to Chemistry.SE! Take the tour to get familiar with this site. Mathematical expressions and equations can be formatted using $\LaTeX$ syntax.For more information in general have a look at the help center. – Martin - マーチン Feb 5 '17 at 9:38
• An $\mathrm{sp^2}$ $\ce{C-H}$ bond is more acidic than an $\mathrm{sp^3}$ $\ce{C-H}$ bond, hence Hc is more acidic than Ha. Allylic protons are a bit more acidic than olefinic protons but they are close, so the final order of increasing acidity would be Ha < Hc <= Hb – ron Feb 5 '17 at 15:18

Hb is the most acidic hydrogen.

When it is extracted a negative charge appears on the carbon attached to it, this negative charge can then undergo resonance with the adjacent double bond. This resonance leads to stabilization of the system. Therefore the compound would prefer losing Hb.

Hence Hb is the most acidic

When comparing between Ha and Hc, Ha would be the more acidic as when we remove Hc there is a vinylic carbanion that is being formed. And Vinylic Carbanions are very unstable. Thus extricating Hb is more feasible thus making it more acidic.

• There is no confusion about Hb being the most acidic. The question is actually asking about which of the other two hydrogens (a and c) is more acidic. – f'' Jul 24 '16 at 16:32
• This answer is incorrect. – Jan Feb 5 '17 at 23:02

To check acidity see conjugate base by removing H+. So when you remove H+ from double bond you get a sp2 carbanion which is more unstable than sp3 carbanion. Now as weaker the conjugate base stronger is acid. hence hc>ha. Hope its clear.

• You have it the wrong way round, the sp2 carbanion is more stable than the sp3 carbanion. – orthocresol Jan 10 '16 at 7:09
• Yes orthocresol is correct , you are wrong . – user101522 Jan 10 '16 at 7:28
• but if we go by conjugate acid base theory the weaker conjugate base means strong acid, isnt it? Can you please explain.. – lollolololol Jan 10 '16 at 11:18


text:
## Friday, June 24, 2011

### hexarideablepod

Rideable Hexapod (aka spiderbot or hexabot or whatever). Anyway, I want.

Step A. KISS. Clone someone else's work on a linkage hexapod.
~$git clone git@instructables.com:Hexabot-Build-a-heavy-duty-six-legged-robot/ haha I wish hardware was as easy as software to duplicate. Substeps: 1. Acquire materials 1. Steel stock 1. Turner Steel (carpool with someone) 1. 60 feet of 1" square steel tubing, 0.065" wall ~$72
2. 20 feet of 2" x 1" square rectangular steel tubing, 0.065" wall ~\$37
3. Directions
4. (508) 583-7800 --inquire about cost for delivery to MIT, also hours
2. Inquire from the Central Machine Shop, bldg 38-001
2. Apply CAD (scale up, can motors handle? torque calcs?)
3. Apply controller to motors (make original one? probably stick with relays)
4. Learn welding (MIG or TIG)
5. Build build build
Oh, right, youtube research: (specifically for this kind of linkage-based rideable hexapod)
Look it's wheelchair motor x2 powered http://www.youtube.com/watch?v=xuRT9gC0CGs

in the meantime, I learned a bit of serial by asking Leighton, a hallmate, so now I understand the pololu code. But I still don't understand why it's sad. Yay I kept thinking it was something with the code, and I couldn't find my cute tiny 2.007 battery to try . But I tried something I knew wouldn't work, and it worked! yay. will post in pololu forums to make it more findable.
http://forum.pololu.com/viewtopic.php?f=16&t=3944&p=19496#p19496