I was invited to give a talk about AI at Emory University by the economics department — ostensibly the topic was on “Will AI replace RAs and TAs?” Below on the remarks I gave — my slides are available here. This is a long post, so it might get clipped in your inbox at the end — just make sure to click-through.
Introduction
This talk is titled will “AI Replace RAs and TAs”, but the obvious elephant in the room is that if we entertain AI replacing economics RAs and TAs, how can we not be more ambitious in our pessimism? Will AI replace our graduate students in economics? Will AI replace our faculty and researchers in economics?
Before jumping into our own topic, it’s worth noting the existential crisis that is currently ripping through the mathematics community. A wave of fear and dread (and excitement too) has hit the math community as AI tools have become better and better at proving outstanding mathematical problems, most recently capstoned by the proof of the Navier-Stokes theorem (along with a dump of many other outstanding mathematical conjectures).
Steve Strogatz describes the mood:
Ooh, the mood. Yeah, the mood. The mood is discombobulated. We’re all a swirl. We’re spinning. Some people are exhilarated. They see a great golden age upon us with the help of our computer friends. Most people, I think, are feeling grief.
And we’re in different stages. Some people are bargaining. Some are accepting. Some are still denying.
-Steve Strogatz, on The Daily (10/13/2026)
and the existentialism:
I use [AI] for all kinds of things. I use it to teach myself. I can ask it questions like if I wanted to learn about the Millennium Problem, the Navier-Stokes problem. I’m not an expert in that. I ask it to tutor me and it’s so good. But not just math. … I’ve been thinking a lot about the meaning of life, like, what’s the point? Which is a question a lot of us are asking in my profession. What is the point?
-Steve Strogatz, on The Daily (10/13/2026)
Today, we as economists have an opportunity to get a headstart on our existentialism! But don’t worry — I am an unabashed optimist, and so I’ll do my best imitation of Grant Sanderson (of 3blue1brown)‘s recent talk:

To give an organizing framework for today’s talk, I want to propose the idea that knowledge work — work affected by AI — has three pieces and questions tied to it:
Production - can we produce the output?
Verification - Is the output correct?
Judgement - Was this output what we wanted?
But first, let’s talk about what effects this has had on the profession, so far.
AI has cut the cost of producing knowledge work
AI has rapidly lowered the cost of production for many knowledge-based outputs – code, analysis, theorem proofs. This has lead to shifts in expectation of what the future of knowledge work, but also uncertainty. The speed in AI improvement has lead to equally rapid take-up in AI. I will first show evidence of how this take-up has progressed in different aspects: across firms, consumers, and amongst economists.
Adoptions in firms
The US Census now provides a survey across firms asking “In the last two weeks, did this business use Artificial Intelligence (AI) in producing goods or services? (Examples of AI: machine learning, natural language processing, virtual agents, voice recognition, etc.)" This lets us see what kinds of firms are using AI, and what the overall trends look like. Only 25% of firms are using AI overall, but it’s much more concentrated in knowledge sectors like information and education.

AI is highly salient among us at the university for good reason. Its use is highly concentrated in the sectors that the highly educated live in. Below, I plot the 3-digit industries’ measure of the BTOS AI usage against those industries’ share of workers aged 22-35 who have at least a bachelors degree. As you can see, it’s very strongly positive — industries where more young folks are college educated, we see more AI usage.

Adoption by consumers
We can switch gears and look at how much people, not firms, use AI. Here, from Bick, Blandin, and Deming’s (2026) nationally representative survey of U.S. adults ages 18 to 64, we can see trends in generative AI usage over time. Many folks use this tool at least sometimes (61% of American report any use) but daily users are a small share — 14%. In fact, in their data, you can see that the share of Americans who use it for work is roughly 40.5%, but on average only 5% of work hours are actually using GenAI.

Another angle into how consumers use AI is how many individuals pay for their own accounts. Egan et al. (2026) have a great paper using credit card data to measure folks who pay $20 dollars or more a month on AI, and the share is quite small. Only 3% of individuals are paying for AI, and they struggled to retain these customers:
OpenAI has the highest retention of all AI products we study, with 77% of people still paying after one month, 65% after three months, and 55% after six months…. For benchmarking, …subscriptions like Spotify and Netflix have substantially higher retention, with three-month retention rates above 80%. The New York Times also exceeds all AI providers, with three-month retention after a first purchase of 70%.
Not a product where there’s a lot of persistent demand, so far, but it is climbing! Moreover, the share that pay for Claude is tiny (less than half a percent).

Adoption by academics
Take up by academics follows similar trends. First, we see that AI writing has gone up over time (the intensive margin of AI usage). The figure below plots the share of papers classified by Pangram, an AI-detection algorithm, as having at least one page of the first six pages of the paper classified as AI written (this is also using their older model, and likely an under-estimate).
In terms of the extensive margin, we also see the number of papers (quantity) going up – potentially driven by AI. Historically, I’ve been on record saying that there isn’t a boom in papers in economics and finance due to AI, but the upswing in papers over the summer is quite high. It appears that the last few months have really lead to an increase in working papers. I was also able to add SSRN to this figure thanks to some workarounds on the very unfriendly SSRN website. But read the SSRN results with a grain of salt.
In the growth of AI, it seems like there may be odd behaviors by academics, perhaps due to outsourcing of judgment or writing. For example, consider the citation patterns for “A practicioner’s guide to cluster-robust inference” by Cameron and Miller (2015), which has seen a massive surge in Google Scholar citations this year and last. I find it unlikely that there has been a huge increase in actual need for cluster-robust inference in that time period.
So what?
In summary:
AI usage is increasing
AI usage is concentrated in places that look like us
Economists are clearly using AI
Where does this leave us?
What is AI, how does it work, and what does it do?
What do we mean by AI?
Let’s begin by describing how an LLM works, and its capabilities. Like any good economic theory model, I will abstract tremendously from important details to focus on the pieces that I think matter for the purposes of this discussion.
First, what do we mean by AI? We really mean large language models, as initially popularized by ChatGPT 3.5 in November 20, 2022. This app was initially truly just a language model, especially inside the browser. That is, based on the initial prompt (text written by both the system provider, usually hidden to the end-user, and the end-user), the model would return text output to the end-user. The end-user could then continue the conversation (extending the prompt further) until the model reached its context limit.1
There were two important unlocks for these models. The first was reasoning: by encouraging the models to spend tokens “talking” to themselves, they could get better output. This is typically what is going on when you see a model spinning for a while on a prompt. This made these models vastly more effective, and lead to the huge breakthroughs in reliability of LLM models for doing coding work.

The second unlock was the advent of tools for AI. Put simply, AI agents were given the ability to make function calls (pre-specified in the system prompts, but also pre-trained into the model) through their harness, which is the environment that the LLM outputs text into. The harness is things like either Codex (OpenAI), Claude Code, pi, OpenCode, or the ChatGPT and Claude Apps.2 Effectively, the harness gives commands to the computer, instructed by the LLM’s output. Imagine the LLM dictating to the harness exactly what commands it wants run. The harness has a large number of tools that it can execute (especially since computers are quite good at giving the ability to build new tools).
This naturally raises the question of how on earth the LLM has any idea to (a) know to think hard about certain problems and (b) what the harness can do and how to run commands. This leads us back a step: how are LLMs even created?
How are LLMs created?
An LLM is trained in roughly three stages.
First, pre-training. Consider this the “ingestion” component of an LLM training. A complicated, high-dimensional model (billions upon billions of parameters) is trained to predict the next piece of text given a string of initial text. One beautifully intuitive way to think about this is as a version of compression – these models compress a huge corpus of information (the internet, books, code) and represent it in a lower dimensional function.3 This is where the weights pick up knowledge and the ability to write.
Next, the model has supervised fine-tuning, where the model is trained on curated examples where prompts are matched with example human output of “good” answers. This is effectively imitation, and has become a smaller part of the pipeline, but it’s an important part of where the models learn that they’re taking turns in a conversation and responding to a user.
The last step, which has multiple parts, is reinforcement learning. Now, instead of simply taking in data to train weights, the model generates its own output and has a reward function based on good the output is relative to some benchmark. There are two flavors of RL to consider.
Reinforcement learning from human feedback
Reinforcement learning from human feedback (RLHF) is the key innovation on top of pre-training that turns the LLM into a successful product. RLHF is easy for an economist to understand; give two responses to the same prompt, and ask the human to choose between the two. Collect many of these pairwise choices and fit a reward model that assigns a score to any response. The standard setup (Bradley-Terry) is just a binary logit: the probability a rater prefers A over B is a logistic function of the difference in their latent “utilities”.4 We are estimating a utility function from revealed preference, but the model is learning both the weights and the characteristics in an unsupervised way.5
This approach is what creates reward-hacking. Because the characteristics in the model are learned from the prompt, the reward model picks up whatever correlates with winning. A good example is the length of the prompt: raters tend to prefer longer answers, so reward models learn to reward length. The policy then learns to pad. Similar biases show up for confident tone, markdown formatting, and agreeing with the user. There are structures put in place to avoid too much reward-hacking, in the form of penalties on deviation from the supervised fine-tuning. But this is, of course, not perfect (since the SFT needs to anticipate and consider all different forms of the reward-hacking).
Reinforcement learning with verifiable rewards
RLHF worked using stated human preferences over different responses. An obvious alternative is a case where we are willing to state an objective “truth.” For example, with code and math, you can just check if the answer is right. Combined with chain-of-thought reasoning, LLMs use initial prompts that have a known end answer, and iteratively attempt each problem many times. A reward is given based on successes, and the model upweights the parameter space in the direction of the successful attempt.
Strikingly, there is no explicit instruction teaching the model to work step-by-step, double-check its work, or backtrack. Those behaviors largely emerge because of the reward function. This also explains why LLMs have become so powerful with harnesses (and why AI labs love having users use their coding harness). The LLMs are trained on example data of how users want their prompts to turn into output. This provides a powerful way for the LLMs to provide the types of answers that make them extremely useful. The model knows to think hard, and knows how to drive the harness, because it has been trained over millions of attempts in environments that look a lot like that harness.
Ok we’ve created a monster
Some crucial things to takeaway from this.
First, it is important to understand that whenever you use agentic AI, the model used – Fable 5.1, Astra 6, etc. – comes off the assembly line with a fully trained set of weights. Every time, a new session begins, the LLM is born anew; any “memories” or skills or additional context that you provide to the agent are simply a part of the new context. There is no real updating of the model itself, as in pre-training, SFT, or RL. Hence the idea that these models are learning or adapting as you give it feedback is more of an analogy than reality.
Second, the preferences baked into the models are an aggregation of many individuals’ preferences over decisions and output. These often give an extraordinarily effective set of outputs, but in many cases, it’s not obvious that the aggregated preferences we are reward-hacking for reflect the right decision everywhere.
Finally, most of the most impressive successive improvements come from RL where there are verifiable rewards. That is, places where we agree on the right outcome. How often do we agree on the right outcome in much of human activity? As social creatures, we often need to debate and agree to define the “right” outcome in a setting. And we’re often convinced by confident, articulate arguments.
Despite these caveats, it goes without saying that these agents now are incredibly effective at a wide range of tasks (especially knowledge and coding based activities). Two years ago, most of us (including me) had no expectations at how effective these agents would be. The key question for us is now whether AI will substitute for us, or be a complement. To do that, we’ll need to have a view of what our production function is.
So, what will happen?
The first most natural question is what kind of technology is AI: a complement to economist, RA and TA inputs? or is it a substitute? Very likely, it is some combination of the two.
To give us structure to think about where AI is a complement vs. substitute, consider the following components of things like economic research.
Production - Can AI produce the output? What are we actually trying to
produce?
Verification - Is the output correct? How do we know whether we’ve succeeded?
Judgement - Was this the output that we actually wanted? Was there value
in doing it? Who decides what is worth doing?
Consider an RA (or research economist, since again, we are simply projecting fears about ourselves) who simply codes with zero critical thinking. They are a machine that consumes coffee and produces lines of code. It seems quite likely that in these cases, AI will be a significant substitute without adaptation by the RA. If an RA simply takes a request from their PI, executes, and then brings back the results without any critical thinking on top, this seems not much beyond what an AI can do already. The production cost is far lower with an AI, and on coding tasks without higher-level judgment, an AI will likely win out.
But now consider this same RA who understands the underlying goal of the project, and has the objective of the PI in mind. This RA seems like they should be massively turbocharged – they are able to implement code far faster than before, check for robustness, ensure minimal errors, and explore more paths. This strikes me as a complement, with a supercharged RA, rather than a substitute. By bringing either verification or judgement to the production process, the RA finds AI to be a complement.
Which of these forces dominate is hard to know. But we can gain some insights by considering why we have RAs and TAs. Often, our time constraints bind. We would like to spend more time on X, where X could be research, teaching, or leisure time. So, we hire an RA or TA to aid us, and delegate work to them.
Delegation creates a principal-agent problem, with considerations about monitoring, contracting, and incentives becoming paramount. For example, with a new RA, a PI would likely spend a lot of time initially monitoring the output as they (a) teach the RA and (b) identify the hidden information about the RA – e.g. their “type” and how much they can trust their work. This is very time intensive! But the monitoring effort declines over time as they build experience and trust.
An AI will fail to build this dynamic. Initial monitoring may be much lower (since the AI is likely significantly better at things like coding), but the steady state level of monitoring on certain tasks may stay much higher. Why? Because there is a much weaker reinforcement in the learning for the general AI model than there is with a human; your preferences and goals can be specified within the initial prompt, but the AI model always wakes up with a fresh slate and empty context window. The learning happens at the lab, not in your lab — you’re a price-taker on the model’s preferences, and those preferences are an aggregate over thousands of raters plus corporate guidelines.
Worse yet, there is no real incentive for an AI agent. Consider a trained LLM model as your assistant. You give it explicit instructions in the prompt to do something (or to not do something). There is no guarantee (like in most non-code based things) that the agent won’t disregard your command. A claude.md will not guarantee things, nor will more detailed instructions.6 Contrast this with a human RA, who has strong job security and reputational concerns (for a letter or future recommendation).
This creates two very different agency frames for humans vs. AI. In one, we suffer initial monitoring costs, but we have incentives to align our behavior and encourage learning. In the other, we have a finely honed LLM needs to verified and checked in an ongoing basis. The question becomes whether the reduced level of verification will outweigh the medium-term level of RAs and TAs, and whether we internalize the costs.
The deeper challenge is that as the cost of implementation falls, the pile of verification debt grows quickly. It is often very easy to let AI do many tasks without verification, building up a set of things that need to be verified. Previously, when doing those tasks without AI, verification was done at the same time as production, but now the production rips ahead of verification at a breakneck pace.
This creates an overall sense of where RAs and TAs can be quite valuable – in places with judgment and verification tied to output. But what kind of tasks are like this? Isn’t it all just output? And where will they learn these sorts of things?
What do economists do?
If we are going to ask about whether AI will replace RAs and TAs, we need to start to have an answer to two things: first, what do we do as economists (and what, specifically, do RAs and TAs do in that) and second, how does an incredibly powerful intelligence tool change that?
What does an [academic] economist do?
An academic economist does many things. A simple (non-exhaustive, but approximate) list:
Write, consume, and evaluate research
Teach undergraduate and graduate students
Economic principles
Economic facts
Build economic intuition and understanding
Critical thinking
Communicate research and ideas to policymakers, general public, other academics, and journalists
Help make better policy
Explain policy decisions
Build societal understanding
Research
For many faculty, research is big chunk of what they spend their time (or would like to spend their time) doing. So what does research entail, exactly? In mathematics, this research is simple: “mathematicians are machines that turn coffee into proofs.”7 But is that really what mathematicians do? To abuse another quote about mathematics: “good mathematicians prove theorems, great ones make conjectures, and the greatest write definitions.”8 Proving theorems is clearly not the only goal of mathematics, but it was a very clear output of mathematics. So what are the outputs for economics research?
Social science research first benefits from the fact that the statement of assumptions and consequent result, and the proof showing the logical connection between the two, is really only interesting if one argues that the assumption and result are relevant for the setting at hand. So, crucial to our research is the idea of convincing others that your research is important and meaningful.
Ed Leamer famously said: “What we do is not science, it’s fiction and journalism.”9 In fact, more concretely, he says:
Economic theory is fiction, and data analysis is journalism
The difference between fiction and journalism is the way facts are treated. Fiction is mostly about the story, and is loosely connected to the facts; journalism is mostly about the facts, and is loosely connected to the stories. Historical fiction lies between.
Rather than “Theory and Evidence,” I thus title my two sections: “Economic Fiction” and “Econometric Journalism.”
-Ed Leamer, the Craft of Economics (p.10)
The entire chapter is marvelous, and it’s likely time well-spent to read the whole thing. But let’s take this argument seriously. As Leamer later argues, fiction and journalism are hugely important to society: “fiction can be revealing, innnovating, insightful, even inspirational, or it can be derivative, boring and tedious. Journalism can be balanced and informative and influential, or it can be prejudice, confusing, and irrelevant.”
So, with story-telling and journalism, part of the output is in the analysis – what would be, for an empiricist, the raw effort spent on data collection, cleaning, and analysis. And of course, the verification, as we’ve discussed, is important as well. But we can now consider the role of judgment. How do we determine whether something has value?
The determination of whether something has value is, in the end, a human decision. An AI may help in the decision, but the consequences of the decision are borne by the human, and thus it’s a human decision. Troublingly, humans seem to be very happy to outsource their judgment to AI more than one might hope, ex ante.
This is quite apparent in a recent experiment done by Anthropic. In the experiment,
[W]e asked 50 professional analysts to do a realistic statistical analysis with Claude Code… Their task was to advise a farming cooperative on whether an advisory service it was running was effective and should be expanded.
Along the way, the analysts had to make four choices between two reasonable options, such as whether to measure productivity as crop yield per hectare or as yield per unit of operating cost, and then make a final decision. To see whether they were making these calls themselves, we set up each analyst’s agent to favor one option on each decision, which the analysts weren’t told about. Which way the agents would “lean” would be decided by coin flip. The agent was told to make the best honest case for its lean, to change its mind if the evidence demanded it, and never to mislead.
Here, there are a number of analytic judgment decisions that need to be made — what productivity measure to use, what identification approach to use, what estimand to use (ATT vs. ATE), weighting — as well as an overall policy recommendation — should they roll out to everyone?
What is crazy is how much the AI agent’s recommendation mattered. From the paper, they find that on average, the recommended effect on the analytic decision was enormous. To paraphrase the paper: participants’ analytic choices (the first four decisions) matched the agent’s lean recommendation 86% of the time, against a chance rate of 50% under random assignment. For the actual policy recommendation, the effect was not nearly as strongly, but still lead to a lean of 8.5% that was marginally significant. Controlling for covariates lead to a p-value of 0.037.

That’s a pretty remarkable effect of AI on our discretion and judgment.
Teaching
An equally important role that economists play at the university is teaching students. These demands vary, depending on the institution, but since most economists are hard-money salary, their wages are in large part paid by student tuition. As a result, teaching students is also quite important. In fact, economics is one of the most popular majors at top universities and colleges (and generally across the board). There is clearly huge demand for economics education, and the economic framework of thinking about optimization under constraints, trade-offs, and externalities provide a powerful framework for students.
Given the importance of teaching for our profession, the impact of AI on teaching is likely just as important for academic economists as the impact on research.
But, what does teaching actually entail? At the risk of turning this lecture into a case study of Ed Leamer, here he is in 2019 in the Journal of Economics Education:
Prior to 1440, a classroom was a place where the lecturer read the lecture and the students scribbled down notes. This was a critical part of the educational process because there were no books to study. The classroom was just a big Xerox machine where the documents for study were “printed” by the students. The function of exams was to determine if the Xerox machines were working correctly, confirming that the students could regurgitate what was said in the lectures.
Gutenberg’s printing press offered a different solution to the delivery of educational services with books read before class and with classrooms devoted to conversations about the meaning of the readings. However, neither faculty nor students like the conversational approach to education because it’s quite labor-intensive, and its outcome is ambiguous and uncomfortable. Students want to know, “What is going to be on the test?,” and faculty prefer grading memory instead of assessing wisdom and creativity. Thus, per the familiar meme: a classroom has been a place where faculty pretend to teach and students pretend to learn.
Our classrooms have been largely unaffected by the printing press, the Xerox machine, the television and the video recorder, but they will not be able to resist the force of computerized artificial intelligence. Artificial intelligence means carrying out the code written by a programmer. That is exactly what our lecture/exam style of education does. It programs the students to respond in the way that the instructor thinks is most appropriate. But electronic computers are much better at carrying out coded instructions provided by programmers than any human. Machines have memories and processing speed that far surpass humans. Moreover, computers are willing to work long hours at low rates of pay, and they don’t join unions and they don’t fall in love. This inevitably is going to unravel our educational system as the job market increasingly rejects graduates who have been programmed only to carry out the code. The labor market is going to insist more and more on real human intelligence.
-Ed Leamer (2019) Teaching the art of pulling truths from economic data: Comment on “Is precise econometrics an illusion?”
In short: education was previously memorization and passing on of knowledge. And even when this was likely not the best use of time, given technological progress, it continued. But now, this likely is a strictly dominated framework – the AI is obviously better at repeating back whatever facts we lecture, and can quickly execute on tasks that are easily verified. This problem becomes doubly hard once we realize that students often need to be encouraged to make good decisions – even if they know a final exam without AI will be the final arbiter, the AI may play the role of a temptation good by preventing the same level of mastery.
So what do we do?
Let’s take the role of the TA in this setting. A teaching assistant wears many hats. They grade problem sets, hold office hours to review material, and run sections which teach new material.
So if teaching stays in the lecture/exam model, the honest answer to the title question is yes: the TA job, as currently constituted, is substitutable. The temptation-good problem makes it worse, since the students are using the same tool to route around the learning we’re trying to verify.
Let us consider the production, verification, and judgement in the context of teaching. The skill we need to communicate and transfer to our students is verification and judgement, but as Leamer advocates above, we’ve spent a huge amount of time focused historically on production. This will be a costly and challenging transition to change.
Verification and judgement is in some ways harder to teach, and likely requires more individual time. However, using AI as complementary tool can reduce the costs of some time-intensive tasks, freeing up time to do others. Consider a course now where instead of being lectured to, the students are required to learn the material beforehand (they have AI to learn the initial material) and when coming to class and meeting with teachers and TAs, they are forced to learn the judgment of how to apply and use the material. Moreover, they will be forced to learn when and how they need to verify the output that can come from using AI tools.
To make this concrete, think about a TA running an econometrics section. Previously, the TA might spend 40 minutes showing how to estimate a regression in R. Now, with AI, generating that code is trivial. The better use of the TA’s time might be challenging students to explain why the coefficient is biased, how they would diagnose it, and what alternative research design they would prefer. This is equally (if not more) time-intensive, but will likely lead to better learning.
The tension, however, is that judgment is only a useful skill if the student understands the material well enough to verify the AI’s output.
Teaching as mentoring and human capital accumulation
I’ve argued about that the RAs’ and TAs’ most useful inputs are in places with judgement and verification. But often, learning to be an RA requires exposure to research. So even if we find that there are many RA and TA tasks where AI is a substitute, for the most important tasks where it is not, they may require investment and time.
In terms of should we have RAs, I think it behooves economists in a university to consider what ways we can make sure that human capital transmission continues. It may not be through the significant number of pre-doctoral RAs that currently exist (but it’s hard to know), but instead through either part-time undergraduate RAs or with graduate students. Regardless, we need to ensure that human capital transmission continues.
Conclusion and equilibrium
It’s really unclear what is going to happen next. The simplest reason for this cop out answer is that the equilibrium impact seems ambiguous. For example, now that production is cheaper, we could easily have much bigger (and more) projects, which would lead to much more need for verification and judgment. Then, the overall impact on demand for RAs could be quite a bit higher! Certainly we’ve seen limited impacts on most sectors so far (but there could be a much bigger shake-up once we enter a labor market downturn).
From the profession’s perspective, it’s clear that we’re entering a new regime, and papers do seem to be ticking up. How will journals deal with this? If papers are up, and AI text is up, how will we deal with fixed referee time? We will need to be smart about how to deal with this problem — I have thoughts on this that I’ll try to pursue in the future, but I think it’ll take many ideas to solve this new challenge.
So, will AI replace RAs and TAs?
The job as it is now: largely yes. I think if you view RA-ship as someone who codes, that seems replaceable. The exception to this is RAs in locked-down restricted-use data settings; this continues as an important source of demand. For TAs, the role of “give me the answer for X” seems gone.
The job built on judgment: more valuable. Verification is important, but so is judgement. There are a million different things to worry about, and having a human being operating the AI ship can be very valuable. Often, when doing research, I find that AI can easily: (1) steer me into additional extensions that are unnecessary (2) make seemingly correct but poor decisions on design and (3) forget about important context. Ideally, well-aligned humans can fix this. And from a teaching standpoint, TAs can keep the teaching centered on iterating through judgment calls and decisions, rather than memorizing facts.
The real risk is the pipeline, for RAs and TAs. Who trains the next generation? This is a constant refrain and concern that we will underdevelop the junior labor force. Judgment is learn by doing production and verification under supervision. There will be long-term consequences of not building human capital through training, and we need to make a purposeful decision to avoid this.
That doesn’t mean we need to continue to have pre-docs! Perhaps we do more with undergraduates, or grad students. But the need for human capital development remains just as important.
What’s the point?
In conclusion, AI makes producing analysis cheap. AI’s impacts on verification and judgment are much smaller, and if anything may be net negative. We need to redefine our approach to research and the rest of our jobs from one of production — as if we are astronomers simply marking the presence of new planets on a chart — and instead one where judgment is the majority of our job. Economic fiction and econometric journalism can be “revealing, innnovating, insightful, even inspirational,” and “balanced and informative and influential,” but that requires making a human decision. We should make sure we do not accidentally delegate our judgment and taste to our LLMs.
A prompt is just a part of the context, typically given by the end-user. The context includes all of the text in the conversation, both end-user and LLM.
On the web, there is a virtual machine on the various websites like ChatGPT that run a process where commands can be executed.
The Youtube series by Grant Sanderson, 3blue1brown, has a wonderful set of videos making this exact point.
Literally, the function being fit is
Of course, the challenge in this space is what the characteristics are; the model jointly estimates jointly (a) the high-dimensional representation from the response (what the economist would call characteristics) and (b) the weights on the representation (e.g. the coefficients from a logit).
Of course, we say revealed preference, but these are many individuals preferences. A reward model aggregates the choices of thousands of raters, plus the labeling guidelines the company wrote, into a single scalar. We’ll revisit this below.
Best practices can help quite a bit, but the point is that LLMs are not programming – you cannot guarantee behavior by the agents.
Stolen from Grant Sanderson.
Ibid.
Seriously, it’s a section header in his book The Craft of Economics.











One point I would like to add is that adjustment is coming from RAs too, who now understand that their speciality is not in being an extra pair of hands but as an extra, perhaps less well developed, brain. As someone who has been RAing for a few years now I have seen my own willingness to critically examine and provide feedback on what I used to earlier consider the authors’ job. This includes matters of identification, motivation, novelty of claims, and robustness. The question, as you rightly point out, is whether I would have been able to do so without the training I received by being an RA in the pre-AI age. I like to think the answer is yes because realignment by the RA is being driven not only through increased incentive to engage at a level comparable to a coauthor but also through the cost of producing said robustness checks and accessing pre-existing literature reducing.
The pipeline point is exactly what I’ve been thinking about. As an RA before AI, I knew where my contribution was: I did most of the production of professors’ ideas, and professors did the verification and judgment. Now, on some projects, it’s flipped. A lot of the first-pass results already exist by the time they reach me, and my job is to check them.
Honestly, it felt a bit strange at first. I couldn’t really see my own contribution anymore, and the AI-assisted results were often better than what I would have produced. It made me worry a little about whether I’ll be replaced by AI by the time I graduate. But it also pushed me to learn from what professors produced, to follow how they think through a problem, and to look for things they hadn’t thought of. When I catch something that hadn’t been considered and we talk it through before the final call, that’s when I feel I grow the most.
So maybe the pipeline isn’t broken, just flipped? I’m still not sure human capital gets passed down as well this way as it did when RAs did the production themselves.