I haven’t had time to write much lately. I spent several weeks in arcane code purgatory, discovering the fun of macros containing uninitialized thread pointers that only fail in 64 bit environments, and for different reasons on Windows, Mac and Linux. That’s a dark place that I hope never again to visit.
Now I’m working fun things again, but they’re secret, so I can’t discuss details. Instead, I’ll just share a little observation that came up in the process.
Frequently, we do calibration or policy optimization on models with a lot of parameters. “A lot” is actually a pretty small number – like 10 – when you have to do things by brute force. This works more often than we have a right to expect, given the potential combinatorial explosion this entails.
However, I suspect that we (at least I) don’t fully appreciate what’s going on. Here are two provable facts that make sense upon reflection, but weren’t part of my intuition about such problems:
- An n-dimensional sphere inscribed in an n-dimensional hypercube occupies approximately none of the volume for large n.
- For random points distributed in n-dimensional space, the distance to one’s nearest neighbor approaches the distance to one’s farthest neighbor for large n.
In other words, R^n gets big really fast, and it’s all corners. The saving grace is probably that sensible parameters are frequently distributed on low-dimensional manifolds embedded in high dimensional spaces. But we should probably be more afraid than we typically are.
That sounds a bit like Michael Betancourt’s description of typical sets in the realm of Hamiltonian Monte Carlo. Thought provoking, isn’t it?
Actually, that may be the context it came up in – differential evolution Markov chain Monte Carlo.