Every day but one in November,¹ I posted a new long-form² article to this site. Some were very successful essays, others were not so good. I’ve started keeping track of my best writing, partly for my own reflection, but also so I know what to recommend to people who want samples of my writing.

Notice that only five of the essays I wrote were really good, by my own opinion. Roughly, one good essay a week. The rest were “everyday essays,” where I didn’t really relate much that was interesting. So from now on, I think I’ll retreat to this, far more manageable, schedule. This means I’ll be able to do research beyond mere Google searches, and I’ll be able to start editing a lot more thoroughly. If you’d like to help read my drafts, and you’re good at proof-reading and critical thinking, I’d be thrilled if you volunteered to help. (Full credit is, of course, given in all the essays.)

Writing more often was a lot of work, and I’m not convinced that daily essay-writing has helped that much. It seems paradoxical, but I think writing slightly less frequently will improve the quality, because it gives me more time for consideration. Certainly towards the end of the experiment, I was exhausted and quite fed up with writing in general, and found it harder to express myself well.³

I’m not the first to note either of these effects: when Aaron Swartz tried it, he came to the same conclusion. He wrote every day for a couple of months, but not on weekends. I think that might be crucial, because my worst work was consistently on Saturday and Sunday. I think I could have done with taking those breaks.

Apart from the 5 essays which I think were very good, there were a few ‘maybes’ — ones which I thought might belong on the list of best essays, but on re-reading didn’t quite stand up to the rest — and the rest were terrible. So there’s a new goal: an essay per week.

I’ve mentioned to a few friends, and publicly on my Colophon, that this website is published with a custom static-file blog engine. Here are the details of how it works.

The main work of producing the site is done by one 200-line PHP file. (It was originally Ruby, but I dislike ERB and the other Ruby template systems so I suffered the pain of rewriting in PHP, just to use Twig.) The actual posts are all plain text files in folders by section (articles, links, photos) and date.¹ There is no database anywhere. The publishing script takes these files, the Twig templates, and various assets (like stylesheets) and uses them to publish the website as a set of HTML files.

(If you’re interested, I have an example of a text file which stores a post.)

The PHP generator script runs on the server. The entire blog is kept in Dropbox, and through a dummy account linked to my server synchs automatically with the server whenever I make a change. Most iPhone and iPad text editors work with Dropbox, so I can write and publish wherever I am.

When the changes on my computers arrive at my server, they’re immediately noticed by inotify, which is attached to a Ruby script, responsible for updating the whole site whenever I make a change (by shelling out to PHP).

This is particularly cool: I have a drafts folder for my unfinished posts. (Including, as I’m writing, this one.) When I save a file there with the Publish: now line in the header, it’s immediately posted publicly, automatically moving to the right place in the directory hierarchy. In addition, cron runs the publisher every five minutes regardless of any changes being made, so I can set Publish to any time in the future and have it automatically pushed live at that time. I’ve only done this once or twice, but it’s quite useful.

One last cool feature is Flickr integration. This took 0 lines of code in the main publisher script; it’s all done by ifttt. Any time I post a Flickr photo publicly with a certain tag, it generates a post for the photos section and automatically puts it in the drafts folder. So (with a 15 minute delay) I can publish my Flickr photos straight to the blog in just two or three steps.

You might notice that it’s not linked to any blog editing interface like MarsEdit. I’d love to hook it up, but I write in BBEdit anyway so the only real advantage would be in post management, for which the Finder is sufficient.

I’ve had a passing interest in areas of philosophy for a long time, but never really known what to do about making it a more serious pursuit. The realm of philosophy seems so big; where should one begin? With the Ancients? The Renaissance? The philosophers of the modern day?

Paul Graham has helped a little with his essay How to Do Philosophy, but I feel like I’m merely scratching the surface of a deeply fascinating discipline.

So I would enjoy a blog offering regular reading recommendations for philosophy, and a guide to the perspectives of the philosophers whose work it recommends.

I’ve received some interesting responses to my post about Philosophy for beginners. I wanted to collate them publicly.

Seth Elliott pointed via Twitter to the Stanford Encyclopedia of Philosophy, which is a little overwhelming as a starting point, but appears to be quite comprehensive as a reference for ideas in philosophy.

More usefully as an introduction, Carl M. Johnson suggested Bertrand Russell’s The Problems of Philosophy, which I’m now reading with some interest. A lot of it seems like the abstract, meaningless stuff Paul Graham complained about in his essay on philosophy as a discipline, but some of it is quite interesting and talks about some unusual ideas.

Speaking of Graham’s essay, Dan Gray responded to it, criticising his arguments. Most of Gray’s rebuttals seem like non-argument arguments, or are only understandable by people already familiar with philosophy. As an inexpert philosopher, I could be wrong, but Gray strikes me as someone who actually believes that the ‘nonsense’ is significant. Whereas I’m persuaded by Graham that philosophy is a superposition of the most abstract concepts in all other fields. Studying philosophy in its own right may be interesting, but only if linked to ideas in other subject areas. Gray argues that meaningless stuff is interesting in its own right. Each to their own — but personally I’d prefer to see how ideas in philosophy can affect ideas in, say, art, mathematics and history, more I want to consider themes in human existence.

Gray’s main problem, I think, is his suggestion that philosophy should remain an abstract mystery. This argument seems like saying that arithmetic shouldn’t be easy to understand. When intellectual groups guard their knowledge in ways that seem deliberately obscure, people distrust that group. For example, I think people don’t trust professional mathematicians and scientists because their ideas seem so hard to understand. For anyone moderately advanced at maths or science, it’s obvious that this can’t really be helped — they really are hard concepts when you don’t have the prerequisite knowledge. But even the easiest areas of philosophy are so obscure that it seems like a joke to most people.

Thanks to all who responded. I’ve a few ideas for essays that I might write up and publish inspired by some of the things I’ve learned.

Education systems are good at producing office workers and physical labourers and bad at making artists and thinkers. Why is this? If you look at how students are treated in school, it’s obvious: teachers are too impersonal. They don’t treat students like their peers as much as they should. They think they belong to an elite group, hierarchically above the students they teach. Many teachers are highly-strung, believe their word is gospel, and don’t accept reasoned argument or tangent to their teachings.

But the best teachers I had were consistently ones who allowed their lessons to divert into tangents, and who expressed their opinions in reasoned debates with their students. And, on the other side, the worst teachers were all dogmatic, didn’t welcome questioning of the ideas they taught, and insisted on acceptance of their doctrines — even when not relevant to the subject they taught.¹

How do we expect students to be good at critical thinking and rational judgement if their daily work environment is hostile to them? Students are told that they’re adults; that they should think for themselves and do what they think is right. Yet when they try to do this, they’re struck down as rebellious.

Worse, this daily inhibition of critical thinking could be destroying students’ creativity. Creative students don’t like school, and tend not to do well in it regardless of their intelligence or talent.² Even the less creative students who are good at mathematics sometimes later find they’re not really good at it, they can just memorise and follow instructions well.

What can be done to fix this? First, getting better teachers: polymaths who can draw on a wide knowledge base. People who are opinionated but pragmatic; who don’t chide students for questioning their ideas; who like working with young people; who have high expectations; and who are so engaged in their subject that they don’t mind being doubted, because they know it might give them a new insight on a concept. Maybe the majority of teachers already there could be like this already, but they’re just afraid to, or aren’t allowed. Certain things, like enthusiasm for the subject, are easy to identify and can’t be faked — and some teachers identifiably have it. Intellectual curiosity is likewise something they can’t beat out of you in teacher training. But other things, like pragmatism, can’t really be faked: everyone thinks they’re open-minded, even if they’re clearly not.

Another aspect of the solution to fix is the stigma surrounding subjectivity in education. Somehow teachers seem afraid to teach things from a subjective standpoint.³ The fear seems to be that opinions will somehow get mixed up with facts, and teachers will mould students into mirror images of themselves, instead of people who can form their own views. This is insulting: it implies that we can’t even distinguish reality from judgment. Just tell us what you think, and we’ll make up our own minds.

In trying to maintain a balanced view, education can actually distort the facts. For instance, climate change, accepted as a fact by scientists, was presented to students as a mere theory, not a real phenomenon: the education system, like the media, was swallowed up by the need to present both views. (Of course, it’s entirely possible that a teacher could believe climate change to be a myth. But I think students could still realise that this was an opinion, and perhaps look at the scientific findings to form their own, correct, belief.)

Finally, curricula should truly encourage independent thought. In theory they already do, but really ‘thinking for oneself’ has become another victim of ‘teaching to the test.’ Students are trained to follow the same thought-paths that examiners take in mark schemes. What good is that: young people trained to think exactly the same as generations prior? Part of the value of youth is willingness (stemming from naïveté and ignorance) to think new thoughts and try new ideas. By Sturgeon’s Law they don’t often work out, but making mistakes is the best way to learn — and by teaching students to unquestioningly think our own thoughts, we stymy their willingness to try new methods and ideas. 90% of the new ideas won’t stick, and the rest might be mostly older ideas, but sometimes the students will strike gold with an entirely new approach.

So the irony is that teachers don’t express their opinions for fear of moulding students’ views and beliefs, but the entirely objective parts of their curriculum do just that. This seems like a recipe for creating a homogeneous thought culture, instead of an intellectual diversity. And indeed the result is that people who want to truly think for themselves often end up in trouble, and the only people who thrive academically are either reserved in expressing their thoughts, or don’t think independently in the first place. Meanwhile, people who have the willingness to try out new things are actively discouraged from doing so. Given that the world is changing so quickly, it seems that trying new things might be the only way to survive — shouldn’t education systems, whose job it is to prepare people for the outside world, be more receptive to this approach?

Thanks to Michael Mokrysz, Isabell Long, and Tim Dobson for reading drafts of this essay.

Further reference.

• Two TED talks by Sir Ken Robinson: Do schools kill creativity? and its follow-up Bring on the learning revolution!
• Paul Graham: What You’ll Wish You’d Known.
• The essays of John Taylor Gatto. (Amusingly, my first introduction to Gatto was a quoted passage in an examination. I don’t know who authorised the quote’s usage, but I bet it wasn’t Gatto himself.)

The world of primality tests is fraught with its own version of the Project Triangle. When you are trying to determine whether a number n is prime, you have three basic options:

• The trial-division method, wherein you try each number up to √n and see if it’s a factor. This is accurate (no number will ever fool this test, because it depends on the very definition of primality) and easy-to-implement, but gets slow quickly.
• Fermat’s primality test. This will be familiar to readers of SICP. It’s easy (once you know the simple theorem behind it) and fast, but not particularly accurate: the pesky Carmichael numbers fool it every time, reporting composite numbers as prime. Despite Abelson and Sussman’s assurance that “the chance of stumbling upon a value that fools the Fermat test is less than the chance that cosmic radiation will cause the computer to make an error,” it’s bad cryptography if an attacker can use a known set of pseudoprimes.
• The Solovay–Strassen and Miller–Rabin tests, which are fast and accurate, but not easy to implement: the former requires understanding the Legendre and Jacobi symbols (arcane mathematical concepts) and the latter requires an efficient factorisation algorithm.

I decided that Plan’s standard library should have a primality test. Not wanting to use the Fermat test because of its inaccuracy, I seemed to be stuck with either the slow trial-division or the complicated Miller–Rabin. (I’m unwilling to clutter the source code of the standard library for the sake of a simple prime number checker.) Fortunately I happened upon an obscure algorithm that seemed to be perfect. It’s called Lehmann’s test.

Here’s the Lehmann algorithm, in four easy steps, for testing a number n:

• Pick a random integer a, 1 ≤ a < n.
• Let x be a^(n-1)÷2 mod n.
• If x is either 1 or (−1 mod n), then n might be prime. Keep trying different random values of a to be sure.
• Otherwise, n is definitely composite — no fooling.

When I understood this test, I was struck by its simplicity and elegance. It works so similarly to the Fermat test, yet (as far as I can tell) is far more accurate.

How do we go about implementing this test, then? Of course, I originally wrote it in Plan, but I’ll present it here in Scheme. We’ll use the same expmod presented in SICP:

(define (square x) (* x x))
(define (expmod base exp m)
  (cond ((= exp 0) 1)
        ((even? exp)
         (remainder (square (expmod base (/ exp 2) m)) m))
        (else
         (remainder (* base (expmod base (- exp 1) m)) m))))

(That’s 1996-style Scheme, of course. You can probably make a few modernisations if you want.)

We also need a random number procedure that’s guaranteed never to give 0, which will cause our test to fail:

(define (random>0 n) (+ 1 (random (- n 1))))

Here’s the test expressed simply in Scheme, where k is our value of certainty (the number of times to run the test) and n is the number we’ll test:

(define (prime? k n)
  (define (lehmann-test tries a)
    (let ((x (expmod a (/ (- n 1) 2.0) n)))
      (cond ((= tries 0) #t)
            ((or (= x 1) (= x (modulo -1 n)))
              (lehmann-test (- tries 1) (random>0 n)))
            (else #f))))
  (lehmann-test k (random>0 n)))

If you try this out in your Scheme interpreter, you’ll notice that it consistently gives errors when you try to find the primality of 1, or of an even number (including 2!). It doesn’t take much effort to fix that, though:

(define (prime? k n)
  (define (lehmann-test tries a)
    (let ((x (expmod a (/ (- n 1) 2.0) n)))
      (cond ((= tries 0) #t)
            ((or (= x 1) (= x (modulo -1 n)))
              (lehmann-test (- tries 1) (random>0 n)))
            (else #f))))
   (cond ((< n 2) #f)
         ((= n 2) #t)
         ((even? n) #f)
         (else (lehmann-test k (random>0 n)))))

According to this page, Lehmann’s algorithm has an overall certainty of 1−2^−k — but that page also gives a bogus definition of the test, so I’d take it with a pinch of salt. But that means that:

• If you run it 10 times, it will run with 99.9% certainty — it’ll be wrong once out of 1024 times.
• If you run it 25 times, it will run with 99.99999% certainty — a 1 in 33,554,432 chance of being wrong.
• If you run it 50 times, it will have 99.9999999999999% (fifteen nines) level of reliability — on average, you’ll have to run it 1,125,899,906,842,624 (a thousand billion)¹ times before it’s wrong.

Don’t forget just how fast this test is. On my machine, testing the prime number 27,644,437 took about 1.2 milliseconds with a k value of 50 (far higher than you’ll probably need for your applications.)

And it’s easy: you could write it on an index card, hand that to an eleven-year-old, and they could probably implement it for you.

As far as I know, there’s no equivalent of the Carmichael numbers for this test. It’s very hard to tell, though, because there’s a dearth of information on this test out there.² But nobody seems to mention that it will report false-positives for any known set of numbers.

Is this test too good to be true? I’d appreciate feedback about its correctness. If it is a perfect prime test (in that, except in cases attributable to its probabilistic nature, it will always produce the correct result for a given input number), I find myself wondering why it hasn’t become more popular.

Updates.

Colin Percival reports that the test is broken for values of n which, as I understand it, are relatively prime to a Carmichael number. I showed, and he explained, that large Carmichael numbers require large values of k for this test to be accurate. So the test isn’t the magic bullet it once seemed, but there seem to be far fewer values for which it gives answers that are outright wrong, compared to Fermat’s test.