NewYorkScapes Archives + Data Projects, 2014-2017

• Municipal Archives Bodies in Transit Records
• Emigrant Savings Bank Test Account Books
• Fales Library Digital Downtown
• NYC NYPL Business Directories

For more see newyorkscapes.org/project

Wilson Business Directory, 1852-53

New York City Directories: Potentials and Pitfalls

• Provide attestations of addresses, including those not on maps
• Occupations and names for the peopled landscape of the city
• Additional attributes such as widowhood of females, marking of "colored"-ness
• Chronological breadth: ~1786-1801, 1849-1922

NYU + NYPL Space/Time: Goals and Objectives

• Build full digital-image-to-Space/Time-data workflow
• Cleaning, ordering, and ingesting "At a Distance"
• Replicable workflows for other city's directories
• Maximize accuracy with minimal human correction

Tesseract and OCR

Doggett & Rode 1851/52 Directory

• Tesseract3 (with training) ➔ Tesseract4 (without training): 10-15% improvement
• OCR error rate of 5% sample with Tesseract4: 11% (with a std dev of 0.07 on post-1849 directories)

A Typology of OCR Errors and Uncertainties

A Typology of OCR Errors and Uncertainties

Goal

From OCR'd text:

Hoffmann Max, physician, 70 Suffolk
Hoffmann Paul, clerk, h 61 Ridge
Hoffmaster Charles A. carver, 220 Centre, h Westchester
Hogan Catharine, wid. John, teacher, 28 James, h 55 James

To a searchable map:

<div class='ocr_page'>
  <p class='ocr_par'>
    <span class='ocr_line' title="bbox 18 679 641 712;">
      Hoffmann Max, physician, 70 Suffolk
    </span>
    <span class='ocr_line' title="bbox 19 712 569 745;">
      Hoffmann Paul, clerk, h 61 Ridge
    </span>
    <span class='ocr_line' title="bbox 21 1200 828 1234;">
      Hoffmaster Charles A. carver, 220 Centre, h West-
    </span>
    <span class='ocr_line' title="bbox 84 1234 562 1259;">
      chester
    </span>
    <span class='ocr_line' title="bbox 862 619 1675 655;">
      Hogan Catharine, wid. John, teacher, 28 James, h
     </span>
     <span class='ocr_line' title="bbox 927 653 1181 679;">
      55 James
     </span>
  </p>
</div>

1. Remove headers, page numbers, etc.;
2. Connect indented lines

const stats = require('simple-statistics')

const xs = page.lines.map((line) => line.properties.bbox[0])

// One indent per column, and four clusters for
//   other stuff like page numbers and headings
const columnCount = 2
const clusterCount = columnCount * 2 + 4

// Find clusters of x coordinates, sort by size
const clusters = stats
  .ckmeans(xs, clusterCount)
  .sort((a, b) => b.length - a.length)

// Compute mode of clusters, and use only
//   the clusters we need
const columns = clusters
  .slice(0, columnCount)
  .map(stats.mode)

console.log(columns)
// [ 34, 840 ]

For all lines that are outside of 2 columns:

1. Find closest line in upper-left direction ↖
2. Connect them if this line is inside column ⟷

const rbush = require('rbush')

// Create an Rbush R-tree index of all line bounding boxes:
const tree = rbush(page.lines.length)
tree.load(page.lines.map((line, index) => {
  const x = line.properties.bbox[0], y = line.properties.bbox[1]
  return {minX: x, minY: y, maxX: x, maxY: y, index}
}))

const knn = require('rbush-knn')

page.lines
  // Only process lines that are not in one of the columns
  .filter((line) => line.columnIndex === undefined)
  .forEach((line, lineIndex) => {
    const lineX = line.properties.bbox[0]
    const lineY = line.properties.bbox[1]

    const filter = (item) => {
      const knnLine = page.lines[item.index]
      const knnLineX = knnLine.properties.bbox[0]
      const knnLineY = knnLine.properties.bbox[1]

      return lineIndex !== item.index &&
        knnLine.columnIndex !== undefined &&
        knnLineX <= lineX && knnLineY <= lineY
    })

    // Find closest 1 line to [lineX, lineY]
    const neighbors = knn(tree, lineX, lineY, 1, filter)

    // If neighbors.length >= 0, we should connect previousLine + line
    const previousLine = neighbors[0]
  })

From:

<div class='ocr_page'>
  <p class='ocr_par'>
    <span class='ocr_line' title="bbox 21 1200 828 1234;">
      Hoffmaster Charles A. carver, 220 Centre, h West-
    </span>
    <span class='ocr_line' title="bbox 84 1234 562 1259;">
      chester
    </span>
  </p>
</div>

To:

{"lines":
  [
    {"text": "Hoffmaster Charles A. carver, 220 Centre, h Westchester"}
  ]
}

• How do we extract structured knowledge from a directory entry?

1. name
   ⇒ "George Binkenburg"
2. occupation
   ⇒ "pianomkr."
3. location predicate
   ⇒ "h." (home)
4. address
   ⇒ "924 Third av."

1. OCR transcriptions are never perfect
   • There are mistranscriptions in the parsed text
   • This makes it very difficult to consistently tokenize the entries

• For instance, consider the (correctly) transcribed entry:
  Binkenburg George, pianomkr. h. 924 Third av.
  • From this, it seems reasonable to split the entry just based on "," and "." characters, and we'd be pretty close!
• Binkenburg George, pianomkr. h. 924 Third av.

• Unfortunately, we rarely see such clean parses!
• Something like this is much more likely:
  • BInkenbu rg Geor ge pianomkr. h 924 Th1rd av

• And not only can we not trust the transcriptions 100%...
• We also can't predict the amount of elements in an entry without inspecting first!
  • Burtnett Daniel, pres. cit. f. ins. co. 67 Wall & 64 Bowery, h. 14 Perry
  • Burt Edwin C. boots & shoes, 43 Broadway, h. 327 Henry, Brooklyn

1. The approach is:
   • Since we can't assume that any delimiter is correctly parsed, we'll first just tokenize everything we can
     • Bush Anna, widow of Henry J. r. 152 W. 13th
     • ⇒ ["Bush", "Anna", ",", "widow", "of", "Henry", "J", " .", "r", ".", "152", "W", ".", "13th"]

• Then, we go through the tokens one by one, and try to figure out what label they belong to
• We can use heuristic methods for classifying each token
  • Does it contain a number? If so, it's probably part of an address.
  • Does it contain a known street name? If so, it's again part of an address.
  • Is it a known occupation? If so, it must be an occupation name.
  • ...

• This turns out to be a somewhat promising approach, but it is very tedious to engineer...
• Our accuracy is contingent on our ability to correctly:
  1. Think of all of the relevant heuristic methods
  2. Interpret and weigh the results of the heuristics correctly

1. We use a supervised-learning algorithm called "conditional random fields" (CRF)
   • Instead of trying to pre-empt what factors are important in labeling, and then interpret the results, we demonstrate by way of labeled examples, and let CRF infer what is important
2. Excellent blog post from NYTimes Developers on CRF

• First we create training examples, at the same individual-token level as before:

• Each token is labeled in terms of what tag it is part of

• This sequence of labels accompanying the tokens is used for training, and also is the form that CRF predictions will take
• Given a sequence of labels, we can then easily parse the record into discrete elements
  • START NC NC D OC OC D PA AC AC PA AC AC END
  • START NC NC D OC OC D PA AC AC PA AC AC END
  • name occupation address 1, address 2

• Conditional random fields uses a discriminative classifier to produce the sequence of labels which maximizes:
  P( word-label | word-features)
• Features encode information relevant to a word's label in a way that is sensitive to the word itself, as well as that word's location in a sequence

• Features are usually boolean (true/false) attributes that are computed for each token
  • In CRF, features describe not only the "present" token, but can also look at adjacent tokens
• We have to create features similar to the heuristic functions we used earlier

• During training time, the algorithm will proceed through the examples, and find the optimal feature weights that maximize the probability of getting the correct label
  • (... this particular task is "convex", meaning there is only one optimum, so we can find it easily)
• This process is done iteratively, often using a training algorithm called stochastic gradient descent
  • (... but luckily for us, we use a library that abstracts this away!)

• Assume that the input we are trying to classify is:
  • Bush Chauncey, commissioner, 157 B'way, h. Rathbun's hotel

• First we'll go through the input, one token at a time, and featurize

• Bush Chauncey, commissioner, 157 B'way, h. Rathbun's hotel
  • current_word-is_start_of_sentence: TRUE
  • current_word-is_end_of_sentence: FALSE
  • current_word-contains_a_number: FALSE
  • current_word-character_capitalized: TRUE
  • ...
  • next_word-is_start_of_sentence: FALSE
  • ...
  • previous_word-is_start_of_sentence: NULL
  • ...

• Bush Chauncey, commissioner, 157 B'way, h. Rathbun's hotel
  • current_word-is_start_of_sentence: FALSE
  • current_word-is_end_of_sentence: FALSE
  • current_word-contains_a_number: FALSE
  • current_word-character_capitalized: TRUE
  • ...
  • next_word-is_start_of_sentence: FALSE
  • ...
  • previous_word-is_start_of_sentence: TRUE
  • ...

• We have a lot of freedom in designing what features to extract
• Features can represent many aspects of the input
• Word shape
• Suffixes and prefixes
• Types of characters in the input
• Whether or not the input is a known entity

• We used Python's sklearn-crfsuite package
  • It conforms to the popular scikit-learn package API
  • It's in Python, so it's straightforward to write custom feature extractors

• What does CRF allow us to do?
  • We just have to create features that are sensitive to differences of tokens
    • But, we don't actually have to determine if those differences are useful or not!
  • CRF infers from training examples (by perceiving in terms of feature vectors) what the most important factors in predicting labels are

• How good is CRF?
  • Super good! Really, really good.
    • But we don't really have many stats to back that claim up
  • Since the majority of our input is unlabeled, it's hard to evaluate against it directly

• You can find our city directory CRF module, as well as our training data, on Github
• We're currently using only about 70 hand-labeled training examples, and this seems to work very well
• Additional tweaks are mostly around edge-cases and infrequent types of statements

{
  "pageNum": 80,
  "bbox": [62, 473, 756, 505],
  "text": "Canvin John, pilot, 309 Water, h. 23 Hamilton",
  "parsed": {
    "subjects": ["Canvin John"],
    "occupations": ["pilot"],
    "addresses":[
      ["309 Water"],
      ["23 Hamilton","h"]
    ]
  }
}

Hamilton Street no longer exists...

We need a database of historical addresses!

Let's use NYPL's digitized maps!

• 861 streets
• 74,635 house numbers

• 861 streets
• 74,635 house numbers
• 48,754 addresses

const normalizer = require('@spacetime/nyc-street-normalizer')

const input = [
  "13th avenue", "Vernon Blvd", "E. 35th", "Macdougal",
  "Shore Rd", "B'way", "E. Twenty-Second St",
]

input.forEach((street) => console.log(street, '⟶', normalizer(street)))
//  13th avenue ⟶ Thirteenth Avenue
//  Vernon Blvd ⟶ Vernon Boulevard
//  E. 35th ⟶ East 35th Street
//  Macdougal ⟶ MacDougal Street
//  Shore Rd ⟶ Shore Road
//  B'way ⟶ Broadway
//  E. Twenty-Second St ⟶ East 22nd Street

Normalized address from city directory:

{
  "pageNum": 80,
  "bbox": [62, 473, 756, 505],
  "text": "Canvin John, pilot, 309 Water, h. 23 Hamilton",
  "normalizedAddress": "23 Hamilton Street"
}

Historical address, with coordinates:

{
  "name": "23 Hamilton Street",
  "data": {
    "mapId":7138,
    "borough":"Manhattan"
  },
  "geometry": {
    "type": "Point",
    "coordinates": [-73.9955835, 40.7110962]
  }
}

• 861 streets
• 74,635 house numbers
• 48,754 addresses
• 3,133,624 city directory entries (28 volumes)
• 967,482 geocoded entries

What's next?

What's next?

Making all this data searchable and visible!