The Structure of New York Financial News

Method

Why alignment is needed. Each decade’s model is trained independently, and independently trained embedding spaces differ by an arbitrary rotation. A raw cosine between two decades would measure that rotation, not any change in meaning. The models are therefore aligned by orthogonal Procrustes onto the 1900s before anything is compared — the method of Hamilton, Leskovec and Jurafsky (2016).

Neighbour filtering. These are fastText models, which build a word from its character n-grams. Left alone, a word’s nearest neighbours are its own spellings and the OCR wreckage around them — equity returns equities, equit, quity. Two filters are applied: a candidate sharing a four-character stem with the query is dropped, and candidates must appear in a 69,905-word English dictionary.

Read the 1890s column with care. Early OCR is markedly worse — 63% of tokens are dictionary words in 1889 against 85% from 1900 — and it shows in the neighbours. In the 1890s crash sits beside cash, brash and cask; ticker beside picker and dicker. These are phonetic and scanning cousins that survive a dictionary filter because they are real words. Part of what the shift measure registers for such words is therefore the corpus getting cleaner, not the language changing. The movements worth trusting are the ones where both ends are substantive — pool from market combination to oil field, curb from the kerbside market to a listed exchange, reserve from a specie proportion to the Federal Reserve.

Comparison with London. The models use the same hyperparameters as the FT models — skip-gram, 100 dimensions, window 5, ten epochs, minimum count ten, character n-grams of three to six — so the two corpora can be read together. Two caveats on that comparison. New York spans four decades against London’s twelve, so distances here are net movement over roughly forty years, not a century, and the magnitudes are not comparable in absolute terms — only the ranking is. And the New York tokenizer keeps two-letter words where London’s dropped them, a small difference in what fell inside each training window.

Method

These metrics are computed by the same module that produced the London site’s, imported rather than reimplemented, so the two are directly comparable: identical word lists, identical denominators, and the same filters — articles under 50 words, or more than 15% digits, are excluded, which drops roughly a quarter of the corpus. Detection is exact whole-word matching against fixed lists, with no parser and no sentiment model, so a metric labelled “moral judgment” counts words like fraud and integrity; it does not understand them.

OCR quality is included as a metric because it is a confound, not a finding. It measures the share of tokens that are common function words, which collapses when scanning fails. Where it dips, treat movement in the other series with suspicion: the paper may not have changed, only the legibility of the scan.

The 1901 spike in Numbers / Article is mostly tables. That series is a raw count, not a rate — the one metric in the set not divided by article length — and it reads 4.50× its baseline in 1901. Normalized per word the same year is only 1.62×, so about two thirds of the apparent surge is articles getting longer rather than the paper printing denser figures. The mean article runs 3,917 characters in 1901 against roughly 1,500 either side of it.

What lies underneath is a real and short-lived editorial habit. Between about 1899 and 1903 the Journal ran long composite tabular compilations — gross-and-net-earnings tables covering every important railroad system, capital-and- dividend tables, monthly comparative statements — and the digitised archive stores each as a single article record. Articles opening with table language (“in the following table will be found…”) are 0.2–0.3% of the corpus in normal years but 3.5% in 1901, a twelvefold jump. They survive the digit filter because the surrounding prose dilutes the figures below the 15% threshold. Numbers / 1,000 Words and Article Length are plotted alongside so the two effects can be separated; the raw metric is kept unchanged because altering it would break comparability with London.

One word list travels badly. Using London’s lists unchanged is what makes the two series comparable, but it imports London’s vocabulary along with them. Personalization counts sir, lord and lady among its eleven terms, honorifics a New York paper had little occasion to print, and the measure duly runs at 1.96 per article here against 4.23 in the Financial Times over the same years. Much of that gap is the word list, not the journalism. The other measures track far more closely — forward-looking 0.575 against 0.572, hedging 0.493 against 0.499 — which is the better evidence that the port is faithful.

What counts as a reference

The criterion is an event a reader would call textbook-worthy, or one later recognised as significant. Five exclusions bind, and they do most of the work: securities named after events do not count (“Dawes bonds” is not a reference to the Dawes Plan); current conditions do not count (an article describing a slump underway is not recalling one); vague gestures to “past panics” do not count; routine corporate events do not count; and nothing inferred rather than read on the page counts.

Denominator. Intensity is references per million words of labeled text — the 71,754 market articles actually scanned, not the full 1.1 million. Raw counts would track how much the paper printed rather than how much it looked back.

Known defects, stated rather than smoothed. Of 21,613 grounded, registry-resolved references in the window, 574 (2.7%) place an event after the article that supposedly recalls it — a registry dating error, since no article can remember what has not happened — and are dropped here rather than displayed. A further 436 references could not be matched to a registry entry and are excluded, and 21 model responses were cut off at a length limit and are incomplete. That leaves 21,039 references on this page. The shaded years at the left rest on very little text — 1889 is 277 articles — so their rates are noise, not trend.

Why this differs from the paper. The published figures are computed on the shared set, the 927 episodes named by both the Journal and the London Financial Times, which is what makes the two comparable. This page uses the full WSJ registry instead, so the counts here are larger and the rates differ: the 1929 Crash runs at 30.1 references per million words over 1889–1930 on this page against 35.8 on the shared set. Both describe the same corpus; they differ in what they divide by.