At 03:34 PM 6/15/00 +0100, Richard wrote:
[...]
> >OK, so we compare the predictions of the two models for the verification
> >experiments. One model does a tremendous job "fitting" the experiment
> >by any quantitative measure, such as square mean error, but does a
> >really lousy job matching the overall "shape" of the data. The other model
> >predicts the shape very well but is way way off quantitatively.
>
>I'm not sure I understand you. How can a curve which gives you a much
>smaller mean square error be a poorer fit to the shape?
OK, lets suppose that the experimental data looks like a segment of a parabola
and is concave up. One model has practically the same shape but is
shifted vertically downward a good bit. This means the square mean error will
be rather large. For the other model, imagine flipping the experimental
data over
so that its concave down. Then superimpose the two as best you can. The
shape will be completely different but the square mean error will be much less
than the other model.
> >What would parsimony have to say in this situation?
>
>Well now you can compare the two models on the basis of predictive success
>*and* parsimony (simplicity). We would hope that the most parsimonious model
>will have had the most predictive success, in which case the choice is easy.
>Otherwise, we have to make a judgment about the relative weight of the two
>criteria.
>
>There have been some significant developments recently in this area of
>statistics, but I'm afraid I'm not up-to-date (I studied statistics over 20
>years ago). For more info, try taking a look at:
>
>http://philosophy.wisc.edu/forster/220/simplicity.html and
>http://philosophy.wisc.edu/simplicity/
Thanks for these links. Very interesting. One thing I neglected to mention
when I asked the question above was that this type of thing (model selection)
is something that I've been doing for many years. In doing what I do, I never
once read any type of philosophy about methodology. I just found things that
work by experience. So, its very interesting seeing what philosophers have to
say about it. How does what I do compare with their suggestions?
Here's something I found right away (from the left column of your second link):
====begin quote==================
The Scientific Method: Section 1.6 is on Occam's razor. The author
claims that "The Razor doesn't tell us anything about the truth or
otherwise of a hypothesis, but rather it tells us which one to test first. The
simpler the hypothesis, the easier it is to shoot down." This is Sir Karl
Popper's view of simplicity, which equates simplicity with falsifiability.
My paper on The New Science of Simplicity argues that this viewpoint
is wrong.
===============================
The part in quotes is exactly what I was going to write as my own view on
Occam. Except the part about easiest to shoot down. From my point of
view that doesn't really enter the equation. The important point from my
perspective is the razor not guaranteeing anything. It gives the direction
of future searching. Why? Because I'm expecting that the best answer is
going to be "simplest" in some way. My problem really is that (a) I cannot see
how we can have any guarantee of this and (b) I cannot see how to quantify
the notion of simplicity, it seems vague and subjective.
The most fascinating part of the quote is Forsters claim that Popper's view
is wrong. So, I've downloaded the paper in which this claim is argued and
hope to read it soon. The implications of the claim are, to me, astounding.
Thus, I am very skeptical right now :).
Another interesting thing I found on the second link is the following:
"However, the aim of model selection should also include the ability of a
model to generalize to predictions in a different domain." -- Forster (from
abstract
to <Key Concepts in Model Selection:Performance and Generalizability>
OK, so I have to really like Forster now :). This is exactly the point I
was trying
to make in my original set up of my question. What I called the "model
verification"
experiment.
OK, let me give a couple of examples I thought of to muddy the waters :).
Perhaps this
will give me time to look at Forster's papers.
Let's take what is supposedly a classic example of parsimony. Copernicus versus
Ptolemy. Which is simpler? Well, Copernicus, obviously. In order to accurately
predict, Ptolemy had to add over thirty deferents and epicycles (circles on
circles).
What could be more absurd? Yet people who tell this story often forget to
mention
(probably they don't know :) that to predict with comparable accuracy,
Copernicus
also required over thirty deferents and epicycles. So, if Copernicus is to
be preferred
over Ptolemy it must be for some reasons other than simplicity and predictive
success.
Now, we all know that Kepler improved this by squishing the circles
slightly into
ellipses. But one can reasonably argue here that parsimony stood in the way of
the advance of science. Kepler wasted a lot of time because he was stuck on
circles. Circles are simpler than ellipses. We also know that Kepler was very
close to the inverse square law and to inertia. Could he perhaps have found
these
had he not been stuck on simplicity? Of course, Kepler's formulation does,
in the
end, turn out to be simpler. But we see this after the fact. The key
element seems
to me to be Kepler's ability to put aside his metaphysical prejudices.
As another example, I will note that we have examples where more than one
explanation
survives. For example, extremum principles versus classical Newtonian
mechanics.
Two ways of looking at the same thing, but yet one has not razored out the
other on
the basis of simplicity. Feynman discusses this in his great little book
<The Character
of Physical Law>:
"One of the amazing characteristics of nature is the variety of
interpretational schemes
which is possible. It turns out that it is only possible because the laws
are just so, special
and delicate." Feynman
This feature I would put under the heading of elegance. But again, like
simplicity, it seems
very subjective.
I guess I have to add one of my favorite Feynman quotes:
"... we have learned from much experience that all
philosophical intuitions about what nature is going
to do fail." -- Richard Feynman
I used to have this in my sig but replaced it by Fats
Waller's version of the same idea :).
Well, enough for now, thanks for your response.
Brian Harper
Associate Professor
Mechanical Engineering
The Ohio State University
"One never knows, do one?"
-- Fats Waller
This archive was generated by hypermail 2b29 : Sat Jun 17 2000 - 21:07:09 EDT