Victim of Expensive Grace (Ken's blog)

Verizon Family and Friends: By the Numbers

2009-05-31T23:03:00.000-05:00

You could guess what your 10 most called are, or you could measure. Guess what I prefer. Find the "Download SpreadSheet" (sic) option on the Verizon site, and pipe it through some Python like this;

import csv, operator
calls = list(csv.DictReader(open('/home/kcarnold/Desktop/VoiceDetails.action'),
delimiter='\t'))
not_in = [c for c in calls if 'M2MAllow' not in c['Usage Type']]
minutes = {}
for call in not_in:
   number, mins = call['Number'], int(call['Minutes'])
   minutes.setdefault(number, 0)
   minutes[number] += mins
sorted(minutes.items(), key=operator.itemgetter(1))

Jesus is Precious

2008-12-10T00:14:00.002-05:00

The apostle Peter says, "To you who believe . . . [he] is precious" (1 Peter 2:7). The grace of God makes Christ precious to us, so that our possessions, our money, our time [and our abilities -ken] have all become eternally and utterly expendable. They used to be crucial to our happiness. They are not so now.

Tim Keller, Ministries of Mercy, page 63 (The Motivation for Mercy: Grace and Generosity)

Sword Project Bible Reader in Python updated

2008-11-01T23:10:00.002-05:00

I've updated and finished the core functionality for a pure Python reader for SWORD Bible modules (hosted on github). I had previously posted an initial proof-of-concept, and now figured out how to map references to keys.

Still to do: OSIS rendering, abbreviation processing, and the sort of convenience stuff that a frontend would want (Next/Previous, etc.).

Supremacy of Christ

2008-08-08T22:12:00.001-05:00

I want today to commend to each one of us the supremacy of Jesus Christ and his Gospel. It is both the supreme glory of God and the solid rock for our lives. These words don't do it justice. It's hard to find any words that do, but these by Piper do a lot better than mine:

http://www.youtube.com/watch?v=oYGLl0gO1dk&feature=related

We were made to know Christ, not do little trivial things.

I was also recently led to read a familiar passage a little differently. When I did, the emphasis lay on these words:

truth, righteousness, gospel of peace, faith, salvation, Word

They are of course the real things that comprise the "armor of God". I encourage you to likewise read Ephesians 6 with the supremacy of the Gospel in view.

One observation I took from that is that one way that the Cross breaks the enemy's power is by removing the allure of the things that he offers. When the enemy offered Jesus bread, he didn't want it because he had something better. If you're concerned about your health, the enemy can control you with something that promises health (like a meditation technique, in my parents' case). But if the Holy Spirit has given you a new heart that regards the love of God as better than life, the temptation of worldly health will have no power over you. In that way, the Gospel (truth, faith, salvation, God's word, etc.) is your armor.

Other observations about the Gospel? Comment away...

Sword Project Bible Reader in Python

2008-08-06T21:04:00.003-05:00

Update: I've updated this and put in on GitHub.

The SWORD API is complex. I did at one point try the SWIG wrapper, but that was still quite difficult to get right.

Fortunately, as complicated as the code is, what it actually does turns out to be very simple. With the help of zverse.cpp, libtool gdb examples/cmdline/lookup, and the Holy Spirit, I came up with this simple Python file to read a verse given an index:


#!/usr/bin/env python

# * ztext format documentation
# I'll use Python's struct module's format strings.
# See http://docs.python.org/lib/module-struct.html
# Take the Old Testament (OT) for example. Three files:
#
#  - ot.bzv: Maps verses to character ranges in compressed buffers.
#    10 bytes ('<IIH') for each verse in the Bible:
#    - buffer_num (I): which compressed buffer the verse is located in
#    - verse_start (I): the location in the uncompressed buffer where the verse begins
#    - verse_len (H): length of the verse, in uncompressed characters
#    These 10-byte records are densely packed, indexed by VerseKey 'Indicies' (docs later).
#    So the record for the verse with index x starts at byte 10*x.
#
#  - ot.bzs: Tells where the compressed buffers start and end.
#    12 bytes ('<III') for each compressed buffer:
#    - offset (I): where the compressed buffer starts in the file
#    - size (I): the length of the compressed data, in bytes
#    - uc_size (I): the length of the uncompressed data, in bytes (unused)
#    These 12-byte records are densely packed, indexed by buffer_num (see previous).
#    So the record for compressed buffer buffer_num starts at byte 12*buffer_num.
#
#  - ot.bzz: Contains the compressed text. Read 'size' bytes starting at 'offset'.
#
#  NT is analogous.

# Configuration (set this to your own modules path):

modules_path = '/home/kcarnold/.sword/modules/texts/ztext'


import struct, zlib
from os.path import join as path_join

class ZModule(object):
   def __init__(self, module):
       self.module = module
       self.files = {
           'ot': self.get_files('ot'),
           'nt': self.get_files('nt')
           }
 
   def get_files(self, testament):
       '''Given a testament ('ot' or 'nt'), returns a tuple of files
       (verse_to_buf, buf_to_loc, text)
       '''
       base = path_join(modules_path, self.module)
       v2b_name, b2l_name, text_name = [path_join(base, '%s.bz%s' % (testament, code))
                                        for code in ('v', 's', 'z')]
       return [open(name, 'rb') for name in (v2b_name, b2l_name, text_name)]

   def text(self, testament, index):
       '''Get the text for a given index.'''
       verse_to_buf, buf_to_loc, text = self.files[testament]

       # Read the verse record.
       verse_to_buf.seek(10*index)
       buf_num, verse_start, verse_len = struct.unpack('<IIH', verse_to_buf.read(10))
     
       uncompressed_text = self.uncompressed_text(testament, buf_num)
       return uncompressed_text[verse_start:verse_start+verse_len]

   def uncompressed_text(self, testament, buf_num):
       verse_to_buf, buf_to_loc, text = self.files[testament]

       # Determine where the compressed data starts and ends.
       buf_to_loc.seek(buf_num*12)
       offset, size, uc_size = struct.unpack('<III', buf_to_loc.read(12))

       # Get the compressed data.
       text.seek(offset)
       compressed_data = text.read(size)
       return zlib.decompress(compressed_data)

if __name__=='__main__':
   import sys
   mod_name = sys.argv[1]
   testament = sys.argv[2]
   index = int(sys.argv[3])
 
   module = ZModule(mod_name)
   print module.text(testament, index)

This was a one-evening project, mostly taken up by the reverse-engineering. As you can see, Python provides a lot of the foundation work so that the code we actually have to write is very small.

I still need to figure out how human-readable verse identifiers get mapped to those numerical indices. It's hidden somewhere in VerseKey...

For everyone who has been worrying about me...

2008-07-23T10:36:00.002-05:00

No need to worry. The worst was really only about a day and a half, and it's been gradually and surely getting better since then. The rash is still there, but it's gone down a lot and now only hurts just enough to remind me that it's there.

So I add a third point to my previous post: 3. How quickly we forget. With the pain basically gone, so is my passion about those two points. It was very real then, but now it's a distant memory. I'm reminded of the stones of remembrance that the Israelites took from the Jordan as they were crossing over; even then, how quickly they forgot! (See Joshua 4:3-7)

Gospel notes from having shingles

2008-07-20T13:56:00.008-05:00

After my mysterious rash turned excruciatingly painful last night, I got it checked out this morning. Turns out it's a classic presentation of shingles. But I'd had it for more than a week before starting medication today, so it's quite painful. In fact, I think it's the first thing that I've experienced that makes me want to say, "it hurts like hell!" And thus I realized:

Praise GOD that although I deserve to endure this for eternity, thanks to Jesus's work I don't have to!

And if your eye causes you to sin, tear it out. It is better for you to enter the kingdom of God with one eye than with two eyes to be thrown into hell, ‘where their worm does not die and the fire is not quenched.’ (Mark 9:47-48)

Then the king said to the attendants, ‘Bind him hand and foot and cast him into the outer darkness. In that place there will be weeping and gnashing of teeth.’ For many are called, but few are chosen.” (Matthew 22:13)

4 Surely he has borne our griefs
and carried our sorrows;
yet we esteemed him stricken,
smitten by God, and afflicted.

5 But he was wounded for our transgressions;
he was crushed for our iniquities;
upon him was the chastisement that brought us peace,
and with his stripes we are healed.

6 All we like sheep have gone astray;
we have turned every one to his own way;
and the LORD has laid on him
the iniquity of us all. (Isaiah 53:3-6)
I want my attitude toward my sins be more like my attitude toward this virus.

This virus is a tiny, otherwise insignificant pest. But I want to kill it! I want it out, never to return.

Oh, that I would view my sins the same way! I'm usually content to let them hide in me; they're small and insignificant and I have more important things to be concerned about. But if I could just see them for what they really are -- painful ruptures in my soul that steal my joy away from God and maul God's dwelling place -- then I'd see how good it is to rip them out!

For if you live according to the flesh you will die, but if by the Spirit you put to death the deeds of the body, you will live. (Romans 8:13)

God may be gracious and bring even more observations to my attention; he's actually shown me a lot in this and I praise him for it (though sometimes with gritted teeth!). Jesus, let me not waste my suffering.

More than that, we rejoice in our sufferings, knowing that suffering produces endurance, and endurance produces character, and character produces hope, and hope does not put us to shame, because God's love has been poured into our hearts through the Holy Spirit who has been given to us. (Romans 5:3-5)

2008-07-05T22:44:00.002-05:00

From Tablet ignites debate on messiah and resurrection:

"His mission is that he has to be put to death by the Romans to suffer so his blood will be the sign for redemption to come," Knohl said. "This is the sign of the son of Joseph. This is the conscious view of Jesus himself. This gives the Last Supper an absolutely different meaning. To shed blood is not for the sins of people but to bring redemption to Israel."

Very interesting article. The true Gospel is that Jesus's death accomplished both atonement for the sins of the people and by that, redemption for Israel. (i.e., those who are, like Isaac, children of the promise, and from every race, nation, tribe, and tongue.)

In short, the "new" discovery just confirms what we've known all along: "that Christ died for our sins according to the Scriptures" (1 Corinthians 15:3).

Django mini-series

2008-06-20T14:30:00.004-05:00

I'm running a Django mini-series on Will Larson's blog, called Wielding Django. So far I have written:

Minimalism

Up to Speed

JSON, Object-Oriented Views, and Starting a Real App

I'm mainly posting this to provide a place for discussion in the comments.

mp3rename

2008-06-07T14:28:00.000-05:00

I had a sizable collection of sermons that were named very badly (e.g., just the name). I was unsatisfied with all the mp3 renaming tools I found; they wouldn't get the output I wanted, or wouldn't read id3v2 tags, or etc. So, using python-eyeD3, I wrote my own.


#!/usr/bin/env python

import eyeD3, os, sys

default_subs = {
    ':': ' -',
}

bad_chars = ['?','/',':']    

def clean(x, subs=default_subs, bad_chars=bad_chars):
    for a,b in subs.iteritems():
        x = x.replace(a,b)
    for c in bad_chars:
        x = x.replace(c,'')
    return x

def do_rename_file(fname):
    tag = eyeD3.Tag()
    tag.link(fname)

    name = []

    num = tag.getTrackNum()[0]
    if num:
        name.append('%02d' % num)
        name.append('-')
    
    artist = tag.getArtist()
    if artist:
        name.append(clean(artist))
        name.append('-')

    title = tag.getTitle()
    if title:
        name.append(clean(title))

    name = ''.join(name)

    if not title:
        print 'Not renaming %s - not enough tag data' % fname
        return

    name = name+'.mp3'
    
    print 'Renaming "%s" to "%s": (OK,q,sub_name)' % (fname, name)
    x=raw_input()
    if len(x) > 2:
        name = x
    elif len(x) > 0:
        print 'OK, skipping.'
        return

    os.rename(fname, name)

if __name__=='__main__':
    for f in sys.argv[1:]:
        do_rename_file(f)

You can modify the output format by... changing the code. But python is pretty readable.

Sound Bytes

2007-01-18T22:13:00.000-05:00

Here are some "Sound Bytes" that I have used to describe my vision. Many are elaborated elsewhere in this blog, but it helps to have everything together in one place:

I want to be a teacher, not a programmer. I don't just mean that I would prefer academia to industry, though that statement is mostly true. I mean that I want to be able to teach a computer, not program it to follow a set of instructions. Currently computers are not very teachable. Many researchers are focusing on the back-end AI and machine learning algorithms necessary for a computer to learn, but learning from a teacher requires two-way interaction. I want to develop this interaction. That is, I want to approach teaching a computer—what is today called programming—as a task in human-computer interaction.

Today, the user types in cryptic code and the computer responds with error messages. My new view of programming is as a process by which a computer helps a human clarify, communicate, and refine ideas. Teaching how to program should be teaching how to teach. The implications of this new view range from end-user development, to modularization, to correctness, to optimization, to educational technology.

This new view requires changes to how we make computers programmable. Instead of just developing new programming languages, we must develop programming environments that allow ideas and relationships to be manipulated in natural, familiar ways. Instead of using syntax to forbid ambiguity, we should allow the user to represent an idea or concept that has not yet been fully developed and help him/her develop it. Ask the user questions about corner cases and reason about them. Likewise, let the user ask questions (e.g., the CMU Natural Programming group’s Whyline). Wherever possible, show behavior “live” (e.g., MIT's Flogo II). The programming environment should guide the user's reasoning toward a consistent and correct concept of the task, clearly communicated to the computer at sufficient detail for unambiguous execution. Programmers should develop concepts, not code.

I envision starting with the development of an end-user programming environment that works with concepts instead of instructions and allows a managed degree of ambiguity. I have done extensive background research on this topic within the context of end-user programming systems. My literature survey paper, done for a technical writing class last semester, is linked on the sidebar.

Concepts and Relationships

2006-12-05T14:18:00.000-05:00

The Cognitive Dimensions of Notation vocabulary calls what I'll discuss here "closeness of mapping". Translation between programmer thought and program notation is difficult. Less required translation is better. So strive to allow the human to express his/her concepts as close as possible to his/her natural way.

What is the natural way we represent concepts? To answer intelligently I'd have to be a psycologist. But I'm pretty sure that any code is not natural. It becomes second-nature to skilled programmers, but only after extensive training, such that us programmers can think directly in the concepts and even language of certain types of computer code.

First observation: we don't express all our concepts the same way. Yes, eventually everything gets stored as connections and signals between neurons, but I don't interact with my concepts that way, or at least I don't try to explain them that way. To explain a mathematical concept, I may lead you through a bunch of symbols and equations. But if you didn't know what the word "mother" meant, I'd try to explain it in much different ways. Perhaps I could express it as symbols and equations, but I'm not sure that would help you.

So on the micro-scale, there is no one optimal language. In other words, tagline 2: "The solution to the problem of programming is not to develop another programming language." In fact, for the low-level nuts-and-bolts, the languages we have are quite good. (Low- and medium- level languages are coupled with hardware capabilities, so as long as hardware stays basically the same, so does the language. I may bring up some alternate ways of doing hardware, now that parallel is hot, that make us re-think these low-level concepts like sequential execution.) Instead, we need to develop a programming environment in which normal humans can work with concepts and relationships in natural ways and eventually--eventually!--put together code.

Second observation: we care a lot about relationships between our concepts. In some schools of thought, concepts themselves are meaningless; only the interactions between concepts (and between concepts and the real world) are meaningful. For a current example of the stress on relationships, see object-oriented programming. But does "object" equal "concept", and "public interface" equal "interaction"? Yes, in some cases. Aspect-oriented programming is trying to capture the rest of the cases. But both paradigms dip shallowly into the richness of our concepts and their interactions, imposing strictness of hierarchy and syntax. [Interaction-oriented(?) programming is taking the first conceptual steps in the right direction here.] An object IS_A concept. So is an aspect, feature, interaction, whatever else we have these days. Let's just do what we always do when we see regularlity like this: generalize it. It seems that a relationship between concepts is itself a concept, or perhaps the instance of one. Oh, instantiation is a concept too. This is all a bit worrying from the where-do-you-stop point of view, but philosophy has the same problem.

Third observation: Our representations of concepts evolve. We often subconsciously refine how we understand things and communicate them.

I'm going to go ahead and post this, even though it's not really done.

Next: Rigidity, Corner Cases, and Syntax

~Ken

Communication and Clarification

2006-12-05T13:31:00.000-05:00

Currently programming is a process of figuring out how to give the computer exactly what it wants in order to perform a certain task.

I want to redefine programming as a process by which a computer helps a human clarify and communicate ideas.

That bears unpacking:

process
Programming is already far more than edit-compile-run, although most traditional languages tie you to at least 2 of those stages. But when I think of "programming", I think of far more than just coding to the whole end-to-end process (if it does end) from an ordinary person having an idea to it being expressed clearly enough that a computer or other person can act on it. More comments on the other person aspect later [FIXME], but it bears saying now that I'd like a computer to be able to help us clarify what we mean when talking with each other.

The central focus is not the code developed, but rather the interaction between programmer and computer. Hence my tagline: "Programming Is Interaction".

Most human-computer interaction (HCI) research focuses on interaction with pre-built applications, such as OS shells, office applications, and websites. Some precious few have applied HCI concepts to programming, with some amazing results. I've written a whole report on it, which I'll post up here sometime [FIXME]. Summary: MIT Media Lab and CMU Natural Programming are the big ones I've seen. [Please let me know if you see any other big work in the field in academia.]

I'll unpack process a lot more over subsequent entries.

help
Humans aren't born knowing how to program. That is, we don't instinctively know (1) how to think logically and communicate ideas clearly, nor (2) how to express these ideas in a computer system. The computer should do all it possibly can to help. Automatic code completion (and in some cases, generation) is just the begininng; how about helping us with the basic reasoning behind it too?

clarify
Generally we don't start out with a perfect idea of anything. What is a "paragraph", or an "audio track"? I can try to give a definition, but surely in the course of developing the word processor or sound editor I will refine this definition quite a bit. My definition will probably not look much like "a paragraph is a _____." anymore either; it will mostly talk about how a paragraph relates to other concepts, like words and pages. ("Concepts and Relationships" will cover this issue in depth. [FIXME])

My initial definition is practically worthless for laying down application code, since refinements in my thoughts will almost certainly change the structure of the implementation. (Ideally my thoughts have the same structure as the implementation. [FIXME]) Yet errors in my initial characterization of the problem can go unsolved for the entire lifetime of the system I develop.

So the programming environment should help me clarify what I mean by a "paragraph", helping me keep my concept consistent (though forgiving me for being temporarily inconsistent -- see "Rigidity, Corner Cases, and Syntax" [FIXME]) while I refine the structure of my thoughts.

It should ask questions like, "What should happen if the area of the paragraph extends outside the page margin?" -- because I gave it the concept of paragraph area and page margin and it determined that the two could be in conflict. Or "what should happen if the first line indent is negative?".

communicate
Everything makes nearly perfect sense, as long as it stays within the confines of my mind. As soon as it tries to leave, it crashes into reality and comes out a mess. In the process of clarification, my goal is to communicate thoughts clearly. Communicating with a computer is far more demanding than another human, because it probably doesn't share my instinctive understanding of the way the world works. But if I've accurately explained my world to the computer, making my concept understandable to it also makes it more understandable to fellow humans. The number of courses we must take in writing shows how little we start off understanding about clear communication. So there's a lot that the computer can do to help. [FIXME: this section is weak and not clearly differentiated from "clarify". Perhaps they are really the same thing.]

ideas
Not algorithms, structures, or processes. I mostly covered this in "(Big-Picture)".

Next: Concepts and Relationships

~Ken

Problems with Programming (Big-Picture)

2006-12-04T16:47:00.000-05:00

Teaching programming should be teaching teaching.

But it's not.

Programming currently requires much more skill than just communicating information logically. It requires:

translating ideas from concepts and formulations natural to humans to those natural to the machine. [The machine makes very little attempt to work with ideas the way humans do.]
communicating these unnatural concepts in an unfamiliar and cryptic code.
knowing lots of details about how things are done internally. [Programming has slowly been getting better on this count.]

At best, current programming environments provide means to navigate the code and show its operation line-by-line.

Worse, the idea of programming today limits the realm of logical manipulation to just what can be expressed as following a sequence of instructions. There is no concept of a concept, only what to do with it on a low level. What if we could logically manipulate

mathematical expressions - showing each step in a derivation for a textbook, for once!
laws - allowing politicians and normal citizens to explore what the effect of a law is in a certain situation
sciences - finally teaching a computer physics or chemistry, not just how to run the numbers behind it
language - besides the obvious application to translation, a computer being remotely able to manipulate language is a great help to language learners
techniques of engineering or other fields - say that someone has taught the computer the technique of least squares (in general); when I am trying to solve a problem that requires some sort of approximation, it offers the least squares technique to me, complete with how to actually do it and the conditions and assumptions I have to make
documents - beyond grammar checking to does this even make sense (and of course much better grammar checking also, with some help from rational annotation, which I'll get into later -- bug me if I don't)

That's just the beginning of the list. I'd venture to say that the kind of thing that will approach general instruction/teaching of a computer would not be called "programming", though for me the development starts there. I welcome suggestions for better names.

Next: Communication and Clarification.

~Ken

Programming as a Basic Skill

2006-12-04T16:34:00.000-05:00

Too many people lack even basic programming skills:

Politicians try unsuccessfully to reason about how their actions will affect complex systems
Lawyers can't write clearly while being clearly understood
Doctors can't communicate their intuitions about medical conditions
Scientists make grave errors in their statistics
Teachers can't communicate their knowledge to students
and everyone commits logical fallacies regularly

People need to learn how to:

communicate their ideas clearly, and
interpret and influence interconnected behaviors that they can't manipulate directly

These are basic skills, perhaps more basic than arithmetic.

A computer's logic is impeccable, yet it can access, store, and manipulate vast quantities of information. So instructing a computer (classically called programming) is ideal for teaching these concepts.

But not today.

Next: Problems with Programming (Big-Picture)

Intro

2006-12-04T16:15:00.001-05:00

[I'm merging my two blogs; this was the first post on reVision: The process of re-visioning programming as interaction, teaching, and idea refinement]

Welcome!

A friend suggested that I blog about some ideas I've had so I could get comments. So over the next few days I'll be writing about where I see programming and human-computer interaction now and where it should be.

I'm doing this to get some ideas out in the open and get comments. Please be wise about what you do with them.

~Ken