I haven’t read the whole column yet, but let me say that I think this effort is extremely useful. Let me tell you my story.
Earlier in my career I tried to exploit my knowledge of econometrics data science and statistics to do applied work. My statistical education went a bit beyond the standard econometrics core —I took classes in predictive modeling and Bayesian statistics with the stat folks. In the stats departament the focus was really applied, so my hope was that I would likely be able to apply those techniques to new problems.
Now, when I came back to the econ world I found that The applied people were hostile to any innovation that was not implemented smoothly in Stata (“why do you partially pool your fixed effects in a hierarchical model? This is very unconventional” “what’s the point of this Gaussian process? Just do OLS”). Nobody wants to ready the methodological section of your paper.
My hope was that I would be more successful if I shopped among the econometrics people. After all, econometrics is supposed to be about developing methods that are ready to use for economic questions, so probably my applied peers would be more receptive to these. However, attending econometric sessions in conferences was much worse: there was exactly zero focus on explaining why this was useful to solve this or that particular problem, it was all about relaxing this or that particular assumption. It seemed written for other econometricians, never with the users of those techniques in mind. So, the hope to use one these new estimators to do produce a new paper seems hopeless.
A related problem is implementation: even if they show that their estimator are useful, econometrics people rarely any software that can be used by applied researchers. This is not surprising as this is a lot of work and hard to maintain, probably not sufficiently recognized within the Econ world. Consider the big contrast with the work that Gelman and his coauthors have done with the Stan environment for Bayesian statistics.
In that respect, I think the work of new DID literature should be the standard: taking a problem in applied research, showing that the standard method delivers bad results, and underscoring how your new estimator does better; they also have developed great if R packages and tutorials and have a webpage for it.
With that backup, it is much easier to people like me to use such estimators in applied research
Yes I have grappled with my own versions of this, and agree with a lot of your sentiment. I wonder how much in the age of AI the costs of implementation will fall?
And I do think there is more focus now on the Econ side of econometrics vs the mathematical statistics
It's honestly weird how rare Gelman-type researchers are, considering how wildly successful and influential he has been as a "theorist of applied statistics". I think part of the problem is that the incentive structure of academia makes this hard to pull off as a career.
This seems great! One problem I often have as an early-career researcher is that I can't tell what work is important versus what isn't, and an experienced applied econometrician's take should be useful!
The disconnect between technical work and empirical work is, I think, pretty universal across fields--it is certainly true in neuroscience! Part of the issue is incentives and specialization: mathy researchers are people who have specialized in math, and they are evaluated by people who have specialized in math on the basis of how impressive and interesting and novel their math is. But for empirical work, you generally *don't want* super impressive or novel math; you want something relatively simple and robust and well-understood, that is *appropriate for your empirical application*.
There's a related problem, which is 'when you have a hammer, everything is a nail'. Theorists and mathematicians are strongly inclined to view all empirical problems as nails, to be whacked with whichever methodological hammer happens to be their own hobby-horse. If you really needed a screwdriver, well, you should have talked to someone in the screwdriver department!*
LLMs might be helpful in solving this problem. They have read the entire econometrics literature, but unlike a human econometrician they have no reason to push one method over another, or a more complicated method over a less complicated method.
*My own favorite thing to do, back in the academy, was talk my way out of authorship credits by convincing experimentalists not to run the complicated model they thought they wanted, and instead just run this line of R code I scribbled on a napkin. It was win-win because they got a better, clearer paper, and I got to do less work.
I haven’t read the whole column yet, but let me say that I think this effort is extremely useful. Let me tell you my story.
Earlier in my career I tried to exploit my knowledge of econometrics data science and statistics to do applied work. My statistical education went a bit beyond the standard econometrics core —I took classes in predictive modeling and Bayesian statistics with the stat folks. In the stats departament the focus was really applied, so my hope was that I would likely be able to apply those techniques to new problems.
Now, when I came back to the econ world I found that The applied people were hostile to any innovation that was not implemented smoothly in Stata (“why do you partially pool your fixed effects in a hierarchical model? This is very unconventional” “what’s the point of this Gaussian process? Just do OLS”). Nobody wants to ready the methodological section of your paper.
My hope was that I would be more successful if I shopped among the econometrics people. After all, econometrics is supposed to be about developing methods that are ready to use for economic questions, so probably my applied peers would be more receptive to these. However, attending econometric sessions in conferences was much worse: there was exactly zero focus on explaining why this was useful to solve this or that particular problem, it was all about relaxing this or that particular assumption. It seemed written for other econometricians, never with the users of those techniques in mind. So, the hope to use one these new estimators to do produce a new paper seems hopeless.
A related problem is implementation: even if they show that their estimator are useful, econometrics people rarely any software that can be used by applied researchers. This is not surprising as this is a lot of work and hard to maintain, probably not sufficiently recognized within the Econ world. Consider the big contrast with the work that Gelman and his coauthors have done with the Stan environment for Bayesian statistics.
In that respect, I think the work of new DID literature should be the standard: taking a problem in applied research, showing that the standard method delivers bad results, and underscoring how your new estimator does better; they also have developed great if R packages and tutorials and have a webpage for it.
With that backup, it is much easier to people like me to use such estimators in applied research
Yes I have grappled with my own versions of this, and agree with a lot of your sentiment. I wonder how much in the age of AI the costs of implementation will fall?
And I do think there is more focus now on the Econ side of econometrics vs the mathematical statistics
It's honestly weird how rare Gelman-type researchers are, considering how wildly successful and influential he has been as a "theorist of applied statistics". I think part of the problem is that the incentive structure of academia makes this hard to pull off as a career.
Please keep doing this on a regular basis!
This seems great! One problem I often have as an early-career researcher is that I can't tell what work is important versus what isn't, and an experienced applied econometrician's take should be useful!
The disconnect between technical work and empirical work is, I think, pretty universal across fields--it is certainly true in neuroscience! Part of the issue is incentives and specialization: mathy researchers are people who have specialized in math, and they are evaluated by people who have specialized in math on the basis of how impressive and interesting and novel their math is. But for empirical work, you generally *don't want* super impressive or novel math; you want something relatively simple and robust and well-understood, that is *appropriate for your empirical application*.
There's a related problem, which is 'when you have a hammer, everything is a nail'. Theorists and mathematicians are strongly inclined to view all empirical problems as nails, to be whacked with whichever methodological hammer happens to be their own hobby-horse. If you really needed a screwdriver, well, you should have talked to someone in the screwdriver department!*
LLMs might be helpful in solving this problem. They have read the entire econometrics literature, but unlike a human econometrician they have no reason to push one method over another, or a more complicated method over a less complicated method.
*My own favorite thing to do, back in the academy, was talk my way out of authorship credits by convincing experimentalists not to run the complicated model they thought they wanted, and instead just run this line of R code I scribbled on a napkin. It was win-win because they got a better, clearer paper, and I got to do less work.
Wonderful - thanks Paul for doing this!
Thank you for this new Substack. This is a great idea. And best wishes!
Macrosoft