I never really worked with DuckDB until I started using Claude Code, but I also noticed that Claude really likes to use DuckDB and it has been super useful to learn.
Thanks for sharing. I’ve been looking for tutorials like this and just found this article from a Chinese research channel. They translated some of your work and it got 11.3 k reads. 2274 people shared it. Just wanted to let you know there are more people reading and please keep sharing!
What pipeline/instructions do you have set up such that Claude asks you questions or enters plan mode? Did you specify the caching downloads or other design decisions? When it compiles the plan, did it do that on its own or are you specifying this with /compact or something?
Thanks for sharing the correct link! I’ve really enjoyed your articles on Claude Code. Most how-to articles regarding AI agents stink of AI slop, but yours are completely different. I have found them to be high quality and instructive.
Inspired by your approach to building firm-level datasets from EDGAR, I constructed a parallel dataset from EDINET and compared CF sign distributions across Japanese and U.S. listed firms. The contrast was striking — I wrote it up here if you're interested: https://kiyonoriiwata.substack.com/p/comparing-cash-flow-structures-of
The sub-agent approach is the right call here. Spawning a separate context window for the web research step keeps the main conversation clean, which matters a lot once your data pipeline gets complex enough that every intermediate result starts bleeding into later decisions. I have been running a similar pattern for months - one orchestrator, specialist sub-agents for bounded tasks, strict handoffs. The part that keeps surprising me is how much the quality improves just from context isolation, not model capability.
The EDGAR rate limit handling is a good example of something that looks trivial but trips up most pipelines. What does your error recovery look like when a filing is missing or malformed mid-run?
The Gap Inc. ticker fix is such a good illustration of something that usually derails a data project for an hour. One rebranding edge case, cached downloads, re-ran in 43 seconds. That's the part nobody talks about when they describe these workflows.
The point about the database being the deliverable, not the raw files, is something I've seen researchers miss repeatedly. Building a queryable asset you can hand to a coauthor or join to Compustat later is a completely different output than a folder of HTMLs. That framing shift alone is worth the read.
We're building a course called "Master Claude in the Real World" that tries to close exactly this gap, where most people are still operating at Level 0 while posts like this one show what Level 3 actually produces. The SEC EDGAR pipeline here would fit right into what we're covering. Just launched on Kickstarter for anyone who wants a structured path through these workflows: https://shorturl.at/ZrG8p
The scale post is the one I've been waiting for. DuckDB on a remote cluster is going to be a different conversation entirely.
Awesome. Something I noticed is that you always run in Opus 4.6. Any reason why? Could sonnet handle the census data pull for instance? (May be Edgar is a bit more complicated)
I would be shocked if it couldn't. I currently am not burning enough tokens (because of teaching) to hit my limits, and so I usually just use Opus as a default (e.g., I use a bazooka when it's really not necessary).
Really cool. Take a look at this plugin — it’s been a gamechanger for me in projects such as this: https://github.com/EveryInc/compound-engineering-plugin
I never really worked with DuckDB until I started using Claude Code, but I also noticed that Claude really likes to use DuckDB and it has been super useful to learn.
Thanks for sharing. I’ve been looking for tutorials like this and just found this article from a Chinese research channel. They translated some of your work and it got 11.3 k reads. 2274 people shared it. Just wanted to let you know there are more people reading and please keep sharing!
Wow that’s amazing to hear! Thanks for letting me know
What pipeline/instructions do you have set up such that Claude asks you questions or enters plan mode? Did you specify the caching downloads or other design decisions? When it compiles the plan, did it do that on its own or are you specifying this with /compact or something?
Did you have an LLM create the link to the Campbell et al. 2014 paper? The link is to the wrong paper.
Thanks! I must’ve, or just some pure home-grown human idiocy. Will fix!
Here’s the correct link: https://link.springer.com/article/10.1007/s11142-013-9258-3
Thanks for sharing the correct link! I’ve really enjoyed your articles on Claude Code. Most how-to articles regarding AI agents stink of AI slop, but yours are completely different. I have found them to be high quality and instructive.
Inspired by your approach to building firm-level datasets from EDGAR, I constructed a parallel dataset from EDINET and compared CF sign distributions across Japanese and U.S. listed firms. The contrast was striking — I wrote it up here if you're interested: https://kiyonoriiwata.substack.com/p/comparing-cash-flow-structures-of
The sub-agent approach is the right call here. Spawning a separate context window for the web research step keeps the main conversation clean, which matters a lot once your data pipeline gets complex enough that every intermediate result starts bleeding into later decisions. I have been running a similar pattern for months - one orchestrator, specialist sub-agents for bounded tasks, strict handoffs. The part that keeps surprising me is how much the quality improves just from context isolation, not model capability.
The EDGAR rate limit handling is a good example of something that looks trivial but trips up most pipelines. What does your error recovery look like when a filing is missing or malformed mid-run?
The Gap Inc. ticker fix is such a good illustration of something that usually derails a data project for an hour. One rebranding edge case, cached downloads, re-ran in 43 seconds. That's the part nobody talks about when they describe these workflows.
The point about the database being the deliverable, not the raw files, is something I've seen researchers miss repeatedly. Building a queryable asset you can hand to a coauthor or join to Compustat later is a completely different output than a folder of HTMLs. That framing shift alone is worth the read.
We're building a course called "Master Claude in the Real World" that tries to close exactly this gap, where most people are still operating at Level 0 while posts like this one show what Level 3 actually produces. The SEC EDGAR pipeline here would fit right into what we're covering. Just launched on Kickstarter for anyone who wants a structured path through these workflows: https://shorturl.at/ZrG8p
The scale post is the one I've been waiting for. DuckDB on a remote cluster is going to be a different conversation entirely.
Awesome. Something I noticed is that you always run in Opus 4.6. Any reason why? Could sonnet handle the census data pull for instance? (May be Edgar is a bit more complicated)
I would be shocked if it couldn't. I currently am not burning enough tokens (because of teaching) to hit my limits, and so I usually just use Opus as a default (e.g., I use a bazooka when it's really not necessary).
Usually, I now run in a coding harness called pi (https://github.com/badlogic/pi-mono/tree/main/packages/coding-agent) where I specify to use different models for different types of tasks (e.g. Opus for planning, Sonnet for coding, Haiku for scouting).
Thanks! Very helpful. What you (and others) do on this front is a great public service.