The singular value decomposition, and how much of an order book is really there

Everything below is standard published material — textbook linear algebra and papers you can pull up and read for yourself. It is a methods piece, written because the singular value decomposition is the first serious tool most people meet when they point machine learning at market data, and because the step that decides whether the answer means anything is the one usually left out.

Start with a matrix, not a method

Two contracts run through this piece: the front-month NIFTY future on the NSE and the front-month SENSEX future on the BSE. They are the natural pair -- two broad Indian index futures, heavily overlapping constituents, daily returns that track each other almost exactly — and yet different enough underneath that the same procedure run on both gives answers which cannot be laid side by side. That comparison gets a section of its own further down. Start with one book.

Photograph the NIFTY future's order book once a second. The exchange disseminates the five best bids and the five best asks, so each photograph is ten numbers: the resting quantity at each of those levels. Stack an hour of photographs on top of each other and you have a matrix A with 3,600 rows and 10 columns.

Nothing has been modelled yet. No assumption has been made about what drives what. There is just a rectangle of numbers, and the honest question about it is: how many independent things are going on in here?

Ten columns does not mean ten independent quantities. When a large seller leans on the book, depth thins across several ask levels at once. When the session quietens after the opening hour, both sides thicken together. The columns move as a group, and the number of genuinely separate movements is smaller — possibly much smaller — than ten.

The singular value decomposition is the tool that answers that question exactly, for any rectangle of numbers, with no assumptions at all.

What the decomposition actually says

Every real matrix A of size m by n, without exception — square or not, full rank or not, invertible or not — can be written as

where U is m by m with orthonormal columns, V is n by n with orthonormal columns, and S is m by n, zero everywhere except on its diagonal, which holds the singular values

The geometric reading is the one worth carrying around. Any matrix, viewed as a transformation, does exactly three things in sequence: it rotates, it stretches along the axes, and it rotates again. Vᵀ is the first rotation, S the stretch, U the second. There is no fourth kind of thing a matrix can do. A transformation that looks like shearing, squashing and twisting at once is always a rotation, a set of independent stretches, and another rotation.

The columns of V are the directions in the input space that get stretched purely — feed one in, and what comes out is the matching column of U, scaled by σ. Those directions are orthogonal going in, and orthogonal coming out. That is the whole content of the theorem, and it is not obvious: for a general matrix, there is no reason to expect a set of perpendicular directions to stay perpendicular after the transformation. The SVD says one always exists.

In plain terms. A matrix is a machine that takes vectors and returns vectors. The SVD finds the small number of directions the machine really cares about, tells you how hard it pushes along each one, and tells you where each lands. Everything else the machine does is a combination of those.

For our book matrix, V holds ten patterns of book shape — each one a recipe over the ten depth columns. U holds how strongly each shape was present at each moment in time. And S says how much of the hour's total variation each shape accounts for.

Why this decomposition and not some other

A matrix can be factorised many ways. What makes the SVD the one that matters is a theorem, not a convention.

Truncate the sum after k terms:

Eckart–Young–Mirsky. Among all matrices of rank k whatsoever, A_k is the closest to A. Not a good choice, not a defensible heuristic — the provable minimiser. The residual is exactly

Eckart and Young proved it for the Frobenius norm in 1936, in Psychometrika of all places, in a paper about factor analysis in psychology. Mirsky extended it in 1960 to every unitarily invariant norm, which is why the result holds however you choose to measure "close".

In plain terms. If you are going to throw away detail and keep a simplified version of your data, there is a best possible simplification, and this is a recipe for computing it. Nothing else you could invent will beat it.

This is why the singular values are the right thing to look at. Squared, they partition the total variation:

so σ₁²/Σσᵢ² is the share of the book's total movement explained by the single most important shape. That ratio is what people are reporting when they say "three factors explain most of it".

The decomposition also lays bare the four fundamental subspaces at a glance: the first r columns of U span the column space, the remaining columns span the left null space, the first r columns of V span the row space, the rest span the null space. And rank — a notoriously fragile thing to compute directly, since arithmetic rounding turns an exact zero into 10⁻¹⁷ — becomes a question of where the singular values fall off, which is a question you can actually answer on a real computer.

PCA is this, said differently

Principal component analysis and the SVD are not cousins. They are the same computation with different bookkeeping.

Centre each column of A by subtracting its mean, and the sample covariance matrix is C = AᵀA/(m−1). Substituting A = USVᵀ:

The eigenvectors of the covariance matrix are the right singular vectors of the centred data. The eigenvalues are the singular values squared, over m−1. Principal components, factor loadings, empirical orthogonal functions, proper orthogonal decomposition — different fields, different names, one computation.

But do not compute it that way. Forming AᵀA and taking its eigenvalues is the single most common numerical mistake in this area, and it is not a small one. The condition number of AᵀA is the square of the condition number of A. If A is moderately ill-conditioned — which order book depth data reliably is, because adjacent levels are close to collinear — then squaring it destroys accuracy in the small singular values, and the small singular values are exactly the ones you were going to use to decide where signal stops.

Run the SVD on the data matrix directly. Golub and Kahan showed in 1965 how to do this stably by bidiagonalisation without ever forming the cross-product matrix, and every library implementation since descends from that work. In practice this is one line — numpy.linalg.svd, scipy.linalg.svd, LAPACK's dgesdd — and the one line is the correct one.

What the factors look like

The classic result of this shape in finance is Litterman and Scheinkman's 1991 study of the US Treasury curve. Applying the same decomposition to bond returns, they found three factors accounting for the overwhelming majority of variation -- around 98% on their data — and, more usefully, that the three had clean interpretations: level (the whole curve moves together), slope (the short end and long end move oppositely) and curvature (the middle moves against both ends).

That paper set the expectation for everything since, and an order book tends to produce a family that rhymes with it:

The second of these deserves attention, because it is not merely interpretable, it is the one with a documented relationship to price. Cont, Kukanov and Stoikov showed in The price impact of order book events (2014) that short-horizon price changes are, to a good approximation, linear in order flow imbalance — a quantity closely related to that second factor — and that this relationship is considerably more stable than impact measured against trade volume alone. If you are going to reduce a book to a handful of numbers, the imbalance direction has a real claim to being one of them.

I have deliberately not attached percentages to that table. Those numbers depend entirely on the instrument, the tick size, the sampling frequency and the session, and quoting someone else's would be worse than useless. Which leads to the part that actually decides everything.

The question almost nobody asks

You have run the decomposition. The first factor takes 40% of the variance, the second 15%, the third 8%. You are about to write "three factors explain 63% of order book variation" and move on.

Stop. Feed the same procedure a matrix of pure random numbers — no structure whatsoever, generated by a random number generator — and it will also hand you a first factor that looks dominant, a second that looks meaningful, and a picture of decaying singular values that looks exactly like structure. The method never returns "nothing here". It always returns a decomposition, and the decomposition always looks like it means something.

So the question is not what are the factors, it is which factors are larger than pure noise would have produced. That question has an exact answer, and it predates its use in finance by thirty years.

The Marchenko–Pastur law (1967). Take a matrix of N variables observed T times, with entries independent, zero-mean, variance σ². Let q = N/T. As both grow large, the eigenvalues of the sample correlation matrix do not scatter arbitrarily — they converge to a specific distribution, supported entirely between

Every eigenvalue inside that band is consistent with there being nothing there at all. It is the noise floor, and it is a formula, not a rule of thumb.

In plain terms. If you correlate enough things over too little history, you will find strong-looking relationships in data you know to be random -- not because you made a mistake, but because that is what randomness does at scale. Marchenko and Pastur worked out precisely how strong those false relationships get. Anything weaker than their bound, you found by accident.

Note what q does. With the ten book columns and 3,600 snapshots of one hour, q ≈ 0.003, the band closes to almost nothing, and real structure is easy to see. Take the NIFTY 50 constituents instead, with a year of daily returns: q = 50/250 = 0.2, and the upper edge sits near 2.1 times the average eigenvalue — a factor now has to be twice the average before it means anything. Widen to 500 names over the same year and q = 2: there are more series than observations, the sample correlation matrix is singular before you begin, and the band swallows nearly everything.

The number of things you measure and the length of history you have are not independent knobs. Doubling the universe without doubling the history makes the results worse, and nothing in the output announces it.

Laloux, Cizeau, Bouchaud and Potters applied this to equity correlations in a 1999 Physical Review Letters paper, and their finding is the one to remember: on 406 S&P 500 stocks over 1,309 daily returns, roughly 94% of the eigenvalues fell inside the Marchenko–Pastur band. Ninety-four per cent of the apparent structure in the correlation matrix was indistinguishable from noise. The largest eigenvalue — the market itself — stood about twenty-five times above the upper edge, unmistakably real. A handful of others cleared the band. The rest was an artefact of finite data.

Plerou and co-authors reached the same conclusion by the same route on a larger US sample in Physical Review E in 2002, and found that the eigenvectors which did escape the bulk corresponded to identifiable business sectors — the deviations from noise are where the economics lives. Johnstone (2001) supplied the refinement for the top of the spectrum: the largest eigenvalue under the null fluctuates according to the Tracy–Widom law, which turns "clears the band" into a proper significance test rather than an eyeballed comparison.

If you take one thing from this piece: compute the noise band before you interpret a factor. It is three lines of arithmetic and it is the difference between analysis and pareidolia.

Four ways market data breaks the recipe

The theory above assumes things market microstructure data does not provide.

Units decide the answer. SVD is not scale invariant. Put resting depth, counted in contracts and running to hundreds, in the same matrix as the spread, counted in ticks of 0.05 and usually one or two of them, and the depth columns dominate the first factor outright. You will have decomposed your choice of units. Standardise each column, or work from the correlation matrix rather than the covariance matrix — and recognise that this is itself a modelling decision, because it declares every column equally important before any data is seen.

Intraday seasonality masquerades as a factor. Volume and spread follow a pronounced U-shape across the session — heavy at the open, thin through the middle, heavy again into the close — documented since Wood, McInish and Ord (1985) and given a theoretical account by Admati and Pfleiderer (1988). Both Indian index futures trade a single continuous session from 09:15 to 15:30, so the pattern is clean and strong. Run a decomposition across a full day without removing it and the leading factor will be the time of day. It will look magnificent, it will be beautifully stable out of sample, and it will tell you that the market opens and closes. Remove the diurnal pattern first, or work within time-of-day buckets.

Asynchrony destroys correlation at short horizons. Two instruments do not trade at the same instants, and measured correlation falls towards zero as the sampling interval shrinks purely from non-synchronous observation — the Epps effect, reported in 1979 and robust ever since. The NIFTY and SENSEX futures are a clean illustration: on daily closes they are all but the same series, and the correlation you measure between them at one-second sampling is markedly lower, with the thinner of the two books losing more. Nothing about the relationship changed between the two measurements. The sampling did.

Heavy tails wreck the estimator. Sample covariance is a maximum-likelihood estimator under Gaussian assumptions. Financial returns are not Gaussian; their tails are heavy enough that a single extreme print can dominate a covariance entry and rotate a singular vector on its own. The standard responses are shrinkage (Ledoit and Wolf, 2004) or the rotationally invariant estimators built directly on random matrix theory, surveyed at length by Bun, Bouchaud and Potters in Physics Reports (2017).

Two futures, one procedure, two answers

Run everything above on the NIFTY future, then on the SENSEX future, and the temptation is to set the two sets of factors side by side and compare loadings. Resist it. The two books are not the same measurement, and the differences are not cosmetic.

The tick is identical; the grid is not. Both contracts move in steps of 0.05 index points, but the SENSEX sits at roughly three times the NIFTY's index number. The same nominal step is therefore about three times finer as a fraction of the level. A tick is not a unit of anything until you divide it by the price, and two instruments quoted in the same nominal increment can have quite different price grids underneath them.

"Level 3" is not a distance. The five disseminated levels are the five best occupied prices, not five consecutive ticks. Where the spread sits at one tick and every step in front of it carries size, the five levels span five ticks. Where the grid is fine relative to the contract's volatility and resting size is scattered, they can span considerably more. So the third column of the NIFTY matrix and the third column of the SENSEX matrix are not measuring depth at the same distance from the mid — and in both, that distance changes through the session. A factor described as "near levels against far levels" is defined against a ruler that is neither fixed nor shared.

The expiry calendars are two days out of phase. Since 1 September 2025, NSE derivatives expire on Tuesday and BSE derivatives on Thursday: the regulator allowed each exchange one weekly expiry day and the two chose differently. Depth, spread and volume in an index future all carry a pronounced weekly cycle around expiry. Stack several weeks of both contracts into one matrix and among the leading factors you will find the beat between two calendars that do not line up. It will be stable, it will clear any noise floor you care to compute, and it will be a fact about the exchange timetable rather than about markets.

The thinner book has a different noise floor. The two contracts do not carry comparable resting size, and on the thinner one the outer levels are empty or near-empty a good deal of the time. A column that is zero a substantial fraction of the session is not a continuous quantity with a variance worth estimating; it is a mixture of "there is size here" and "there is nothing here", and a sample covariance will not tell you which it just measured. Fewer genuinely active snapshots also means a smaller effective T, hence a larger q, hence a wider Marchenko–Pastur band. The same eigenvalue that counts as structure on the busier book can fall inside the noise band on the quieter one. The procedure did not change; the bar it has to clear did.

What survives the comparison is not the loadings but the structure around them: how many factors clear the noise band, what share of variation the first one takes, and whether the leading subspace is stable from one window to the next. Those are comparable across instruments. The vectors themselves are not, and two factor-three plots drawn side by side are an illustration of nothing.

Two things that look like discoveries and are not

Sign is arbitrary. If (u, σ, v) is a singular triplet, so is (−u, σ, −v). The decomposition is only determined up to sign, and library implementations make no promise of consistency between calls. Re-run your pipeline on tomorrow's data, find factor three has flipped, and you have discovered a convention, not a regime change. Pin the sign yourself — force the largest-magnitude loading positive, or align each vector against yesterday's.

Close singular values are not separately meaningful. If σ₄ and σ₅ are nearly equal, the individual vectors v₄ and v₅ are numerically unstable — any rotation within the plane they span is very nearly as good a decomposition, and small perturbations in the data will spin them freely. The subspace is stable; the individual axes in it are not. Interpreting v₄ on its own in that situation is reading meaning into an arbitrary choice of basis. Wedin's theorem gives the formal statement: the perturbation of a singular subspace is controlled by the gap to the neighbouring singular values.

Computing it at size

Full SVD of an m by n matrix costs O(mn²) for m ≥ n. For 3,600 by 20 that is nothing. For a cross-section of thousands of instruments at high frequency it stops being nothing.

Three practical moves, in order of how often they are the right one:

• Thin SVD. You almost never need the full m by m matrix U. The thin form computes only the first r columns and is what every library gives you by default when asked (full_matrices=False in NumPy). Asking for the full form on tall data allocates an enormous matrix you will not use.
• Truncated and randomised SVD. If only the top k factors are wanted, the randomised algorithms of Halko, Martinsson and Tropp (2011) get them in roughly O(mnk) with rigorous probabilistic error bounds. This is the single biggest practical win available, and it is well-implemented and widely available.
• Incremental SVD. For a decomposition that must track a streaming book rather than be recomputed each bar, Brand (2006) gives rank-one updates to a thin SVD, which is how you avoid redoing the whole thing every second.

What it cannot do

This is the part that separates using SVD from believing it.

It is unsupervised. It finds directions of maximum variance in the inputs. It has never seen your target and has no way to know what you want to predict. Directions of large variance are not directions of large predictive power, and the two can be nearly orthogonal — the leading factor may be a common movement that is priced and unforecastable, while a small factor carries what you are looking for. If you want components chosen for their relationship to a target, that is a different tool: partial least squares, or reduced-rank regression in the sense of Izenman (1975).

It is linear. It finds linear combinations of columns, so it cannot represent a threshold, a saturation, or a relationship that reverses sign in stressed conditions. It also assumes the structure holds throughout the sample. If the book behaves differently in the first ten minutes than in the middle of the day, one decomposition over both produces an average of two regimes that describes neither.

It is second-order. Means and covariances are all it looks at. Skew, tail dependence and the asymmetry between how books thicken and how they evaporate are invisible to it by construction.

It is not causal. A factor is a statistical summary. That book imbalance and price changes load on the same component says they move together. It does not say which one moves first, and the microstructure literature exists in large part because that question is genuinely hard — see Hasbrouck (1991) for the canonical treatment of separating information from mechanics.

How you would know it worked

Four checks, none of which require a trading result to evaluate:

1. Compare against the noise band. Marchenko–Pastur for the bulk edge, Tracy–Widom if you want a real p-value on the top eigenvalue. Factors inside the band are not findings.
2. Test the subspace out of sample, not the vectors. Decompose two disjoint time windows and measure the principal angles between the leading subspaces (Björck and Golub, 1973). Small angles mean the structure persisted. Comparing individual vectors instead will report instability that is only sign and rotation ambiguity.
3. Shuffle. Permute each column independently in time. This destroys the cross-sectional relationships while preserving every marginal distribution, heavy tails included. Re-run. Whatever survives that was never structure.
4. Check the interpretation is forced, not chosen. If the second factor is imbalance, its loadings should be near-antisymmetric between bid and ask columns because the data made them so, not because you rotated until they looked that way. Varimax and its relatives will happily deliver a tidy story from noise.

The honest summary is that the singular value decomposition is exceptionally good at one thing: given a rectangle of numbers, it produces the best possible low-rank description of it, provably, without assumptions, in one call. What it will not do is tell you whether that description is of the market or of your sample. That second question is answerable — Marchenko and Pastur answered it in 1967 — but it has to be asked separately, and it is the one that decides whether any of the rest was worth computing.

Sources

Every empirical figure quoted above is attributed to the paper it came from at the point it is used, so each one can be checked against the original rather than taken on trust. The theorems are standard results carrying their authors' names; the readings of them, the order book examples and the comparison between the two contracts are my own.

The linear algebra:

• G. H. Golub and C. F. Van Loan, Matrix Computations, 4th ed., Johns Hopkins University Press, 2013 — chapters 2 and 8 are the reference.
• L. N. Trefethen and D. Bau III, Numerical Linear Algebra, SIAM, 1997 — lectures 4 and 5 give the clearest short treatment of the geometry.
• C. Eckart and G. Young, "The approximation of one matrix by another of lower rank", Psychometrika 1(3), 211–218, 1936.
• L. Mirsky, "Symmetric gauge functions and unitarily invariant norms", Quarterly Journal of Mathematics 11, 50–59, 1960.
• G. H. Golub and W. Kahan, "Calculating the singular values and pseudo-inverse of a matrix", SIAM Journal on Numerical Analysis 2(2), 205–224, 1965.
• G. W. Stewart, "On the early history of the singular value decomposition", SIAM Review 35(4), 551–566, 1993 — Beltrami in 1873, then Jordan, Sylvester and Schmidt, each independently.
• N. Halko, P. G. Martinsson and J. A. Tropp, "Finding structure with randomness", SIAM Review 53(2), 217–288, 2011.
• M. Brand, "Fast low-rank modifications of the thin singular value decomposition", Linear Algebra and its Applications 415(1), 20–30, 2006.
• Å. Björck and G. H. Golub, "Numerical methods for computing angles between linear subspaces", Mathematics of Computation 27(123), 579–594, 1973.

The noise question:

• V. A. Marchenko and L. A. Pastur, "Distribution of eigenvalues for some sets of random matrices", Mathematics of the USSR-Sbornik 1(4), 457–483, 1967.
• L. Laloux, P. Cizeau, J.-P. Bouchaud and M. Potters, "Noise dressing of financial correlation matrices", Physical Review Letters 83(7), 1467–1470, 1999.
• V. Plerou, P. Gopikrishnan, B. Rosenow, L. A. N. Amaral, T. Guhr and H. E. Stanley, "Random matrix approach to cross correlations in financial data", Physical Review E 65, 066126, 2002.
• I. M. Johnstone, "On the distribution of the largest eigenvalue in principal components analysis", Annals of Statistics 29(2), 295–327, 2001.
• O. Ledoit and M. Wolf, "A well-conditioned estimator for large-dimensional covariance matrices", Journal of Multivariate Analysis 88(2), 365–411, 2004.
• J. Bun, J.-P. Bouchaud and M. Potters, "Cleaning large correlation matrices: tools from random matrix theory", Physics Reports 666, 1–109, 2017.

The market structure:

• R. Litterman and J. Scheinkman, "Common factors affecting bond returns", Journal of Fixed Income 1(1), 54–61, 1991.
• R. Cont, A. Kukanov and S. Stoikov, "The price impact of order book events", Journal of Financial Econometrics 12(1), 47–88, 2014.
• J. Hasbrouck and D. J. Seppi, "Common factors in prices, order flows, and liquidity", Journal of Financial Economics 59(1–2), 383–411, 2001.
• J. Hasbrouck, "Measuring the information content of stock trades", Journal of Finance 46(1), 179–207, 1991.
• T. W. Epps, "Comovements in stock prices in the very short run", Journal of the American Statistical Association 74(366), 291–298, 1979.
• R. D. Wood, T. H. McInish and J. K. Ord, "An investigation of transactions data for NYSE stocks", Journal of Finance 40(3), 723–739, 1985.
• A. R. Admati and P. Pfleiderer, "A theory of intraday patterns: volume and price variability", Review of Financial Studies 1(1), 3–40, 1988.
• J.-P. Bouchaud, J. Bonart, J. Donier and M. Gould, Trades, Quotes and Prices: Financial Markets Under the Microscope, Cambridge University Press, 2018.
• M. D. Gould, M. A. Porter, S. Williams, M. McDonald, D. J. Fenn and S. D. Howison, "Limit order books", Quantitative Finance 13(11), 1709–1742, 2013.

Disclaimer and disclosures

This is not investment advice. It is a methods piece about a technique in linear algebra. Nothing above is a recommendation to buy or sell anything, a trading strategy, a signal, or a view on the direction of any instrument or index. It takes no account of anyone's objectives, financial situation or needs.

No claim is made that any of this makes money. No performance figure, no back-test and no track record appears anywhere in it, deliberately. A method that separates structure from noise is a measurement tool; measuring something accurately is not the same as being able to profit from it, and this piece makes no assertion about the second.

The figures illustrate, they do not report. Two of them are exact computations: the ellipse is the true image of the unit circle under the matrix printed beside it, and the density curve with its two edges is the Marchenko–Pastur law for the stated ratio. The three factor shapes are labelled schematic on their face and show the form of a set of loadings, not measurements from any book; the eigenvalue marks along the axis of the density figure, and the weeks in the expiry diagram, are illustrative in the same way. No proprietary, licensed or exchange-supplied data was used to produce any of them, and none is reproduced anywhere in this post.

Sources are other people's work, and are cited as such. Every empirical finding quoted is attributed at the point it is used, with the journal and year, so it can be checked against the original. The theorems carry their authors' names. The exposition, the order book examples, the comparison between the two contracts and any errors in them are mine.

Trademarks belong to their owners. NIFTY and NIFTY 50 are trademarks of NSE Indices Limited and the National Stock Exchange of India; SENSEX is a trademark of BSE Limited. They are named here descriptively, to discuss publicly known contract mechanics. EdgeQuant Trading is not affiliated with, sponsored by or endorsed by either exchange or any index provider, and nothing here is issued with their involvement or approval.

Standing disclosures. EdgeQuant Trading is not a registered investment adviser or research analyst, does not manage money for others, does not publish signals, and does not distribute trading systems. No payment, sponsorship or other consideration was received in connection with this post, and no product, vendor, broker or data provider mentioned or implied has any commercial relationship with it. Contract specifications and exchange timetables were correct as described when this was written; they are set by the exchanges and the regulator and do change, so verify them at the source before relying on any of it.

Corrections are welcome. If something here is wrong, say so and it will be fixed with the change noted.