Show HN: Use Claude Code to Query 600 GB Indexes over Hacker News, ArXiv, etc.

Exactly, people want precision and control sometimes. Also it's very hard to beat SQL query planners when you have lots of material views and indexes. Like this is a lot more powerful for most use cases for exploring these documents than if you just had all these documents as json on your local machine and could write whatever python you wanted.

Yeah I've out a lot of care into rate-limiting and security. We do AST parsing and block certain joins, and Hacker News has not bricked or overloaded my machine yet--there's actually a lot more bandwidth for people to run expensive queries.

As for getting good semantic queries for different domains, one thing Claude can do besides use our embed endpoint to embed arbitrary text as a search vector, is use compositions of centroids (averages) of vectors in our database, as search vectors. Like it can effortlessly average every lesswrong chunk embedding over text mentioning "optimization" and search with that. You can actually ask Claude to run an experiment averaging the "optimization" vectors from different sources, and see what kind of different queries you get when using them on different sources. Then the fun challenge would be figuring out legible vectors that bridge the gap between these different platform's vectors. Maybe there's half the cosine distance when you average the lesswrong "optimization" vector with embed("convex/nonconvex optimization, SGD, loss landscapes, constrained optimization.")

This is the route I went for making Claude Code and Codex conversation histories local and queryable by the CLIs themselves.

Create the DB and provide the tools and skill.

This blog entry explains how: https://contextify.sh/blog/total-recall-rag-search-claude-co...

It is a macOS client at the present but I have a Linux-ready engine I could use early feedback on if anyone is interested in giving it a go.

I don’t have the experiments to prove this, but from my experience it’s highly variable between embedding models.

Larger, more capable embedding models are better able to separate the different uses of a given word in the embedding space, smaller models are not.

> I like that this relies on generating SQL rather than just being a black-box chat bot.

When people say AI is a bubble but will still be transformational, I think of stuff like this. The amount of use cases for natural language interpretation and translation is enormous even without all the BS vibe coding nonsense. I reckon once the bubble pops most investment will go into tools that operate something like this.

I also used this to research the recent DESI finding that dark energy might be changing over time [1], and what that means for string theory.

From Claude:

  Summary of Research Findings

  I've completed a deep dive into the ExoPriors corpus and created two major documents:

  1. /research/exopriors_findings.md - GA and String Landscape Papers

  Key discoveries:
  - arXiv:1907.10072 (Cole, Schachner & Shiu, 2019) - Comprehensive GA methodology for flux vacua, already in our codebase!
  - arXiv:1302.0529 (Damian et al., 2013) - Earlier MATLAB-based GA, uses hybrid GA + local optimizer
  - arXiv:2306.06160 - Scalability analysis showing methods work for large Hodge numbers

  2. /research/DESI_DARK_ENERGY_IMPLICATIONS.md - DESI Dark Energy Impact

  Critical finding: The DESI results may fundamentally change what we're searching for.

  The DESI result in brief:
  - Dark energy equation of state w ≈ -0.55 (not -1)
  - Evidence for evolution: wₐ ≈ -1.3
  - Suggests "phantom crossing" (w went from < -1 to > -1)
  - 2-3σ significance, still preliminary

  Key papers found:
  1. arXiv:2112.10779 (Cicoli et al.) - Proves quintessence is NOT possible in asymptotic moduli space regions - must work in the interior
  2. arXiv:2511.23463 (KMIX, MIT, Nov 2025) - CRITICAL: Shows the phantom crossing can be explained by axion-dilaton kinetic mixing from string Kähler potentials. The "phantom" is an illusion!

  What this means for our project:
  - Instead of targeting static Λ = 2.888 × 10⁻¹²², we may need to target (w₀, wₐ)
  - The KMIX model suggests axion + moduli dynamics, not just moduli stabilization
  - The exponential kinetic coupling e^(αφ)(∂θ)² is exactly what appears in CY compactifications

  Next steps:
  1. Study KMIX model in detail - it uses string-motivated physics
  2. Consider adding (w₀, wₐ) to our fitness function
  3. Possibly include axion dynamics alongside Kähler moduli
  4. Update FORMULAS.md with quintessence equations

I just built something like this a week ago: https://github.com/eamag/papers2dataset

I wanted to find all cryoprotective agents that were tested at different temperatures, but it should be extandable to your problem too. Uses OpenAlex to traverse a citation graph and open access pdfs

Today I finally got Claude working in a devcontainer, so I'm wondering what the easier options are.

Exactly. Prompt + Tool + External Dataset (API, file, database, web page, image) is an extremely powerful capability.

It's just marketing.

It is not a protected term, so anything is state-of-the-art if you want it to be.

For example, Gemma models at the moment of release were performing worse their competition, but still, it is "state-of-the-art". It does not mean it's a bad product at all (Gemma is actually good), but the claims are very free.

Juicero was state-of-the-art on release too, though hands were better, etc.

The scale. How many tools do you know that can query the content of all arxiv papers.

in the direction of "empowering the public with new capabilities they didn't have before", Scry offers, with the copy and paste of a prompt and talking with an agent:

1) Full readonly-SQL + vector manipulation in a live public database. Most vector DB products expose a much narrower search API. Basically only a few enterprise level services let you run arbitrary SQL on remote machines. Google BigQuery gives users SQL power, but it mostly doesn't have embeddings, connect public corpora, have as good of indexes, and doesn't have support an agentic research experience. Beyond object-level research, Scry a good tool for exploring and acquiring intuitions about embedding-space.

2) An agent-native text-to-SQL + lexical + semantic deep research workflow. We have a prompt that's been heavily optimized for taking full advantage of our machine and Claude Code for exploration and answering nuanced questions. Claude fires off many exploratory queries and builds towards really big queries that lean on the SQL query planner. You can interrupt at any time. You have the compute limits to do lots of exhaustive exploration--often more epistemically powerful than finding a document often, is being confident than one doesn't exist.

3) dozens of public commons in one database, with embeddings.

The tool is state of the art, the sources are historical.

First, so best in this?

What is hyperbole? We are collectively experiencing a software intelligence explosion (people are shipping good software at prolific rates now due to Opus 4.5 and GPT-5.2-Codex-xhigh). With Scry, you can run arbitrary SELECT SQL statements over a large corpus and have an easier time composing embedding vectors in whatever mathematical ways you want, than any other tool I've seen.

I could make it open-source as soon as I have $5k to my name. I've been in survival mode frankly for a long time.

If you're not willing to pay for your own LLM usage to try a free resource offered by the author, that's up to you. But why complain to the author about it? How does your comment enrich the conversation for the rest of us?

It's ultimately just a prompt, self-hosted models can use the system the same way, they just might struggle to write good SQL+vector queries to answer your questions. The prompt also works well with Codex, which has a lot of usage.

I think that’s just a matter of their capabilities, rather than anything specific to this?

I mean I've been living off of $1700/month for a while in Berkeley. I have been trying hard the last 6 weeks to raise angel investment, and am moving to Thailand in a few days to have more breathing room (and change things up to untie some emotional knots and try to make sure I'm positioned to vibe-engineer as well as possible over the next few months).

Thank you, I've started ingestion operations of pubmed.

I think you misunderstood. The API key is for their API, not Anthropic.

If you take a look at the prompt you'll find that they have a static API key that they have created for this demo ("exopriors_public_readonly_v1_2025")

Thank you!

Claude Opus 4.5 is a paradigm shift

Thank you! I got the idea December 3, and initially released it December 19.

Would you mind walking through the logic of that a bit for me? I'm definitely interested in productizing this, and would be interested in open sourcing as soon as I have breathing room (I have no money).

Should be squared away now! Was my fault missing a health check for a recent weird bug, not a load issue.

Great idea! I just overhauled the prompt to explain the SQL + semantic filters better, and give the user clearer adjustment opportunities before long-running queries.

That would be amazing! Yes, contact@exopriors.com.

Thank you! I'll be getting millions more quality, embedded documents, it'll be here just getting more useful.

The use case could vary from person to person. When you think about it, hacker news has large enough data set ( and one that is widely accessible ) to allow all sorts of fun analyses. In a sense, the appeal is:

who knows what kind of fun patterns could emerge

You submit a SQL query to periodically run, we run it and store the results. As we ingest more documents (dozens of sources are being ingested every day), we run it again. If there's different outputs, you get an email.

I've since improved it, and also discovered a new method of vector composition I have added as a first-class primitive:

debias_vector(axis, topic) removes the projection of axis onto topic: axis − topic * (dot(axis, topic) / dot(topic, topic))

That preserves the signal in axis while subtracting only the overlap with topic (not the whole topic). It’s strictly better than naive subtraction for “about X but not Y.”

Just comes down to your own view of what AGI is, as it's not particularly well defined.

While a bit 'time-machiney' - I think if you took an LLM of today and showed it to someone 20 years ago, most people would probably say AGI has been achieved. If someone wrote a definition of AGI 20 years ago, we would probably have met that.

We have certainly blasted past some science-fiction examples of AI like Agnes from The Twilight Zone, which 20 years ago looked a bit silly, and now looks like a remarkable prediction of LLMs.

By todays definition of AGI we haven't met it yet, but eventually it comes down to 'I know it if I see it' - the problem with this definition is that it is polluted by what people have already seen.

I want to know what the "intelligence explosion" is, sounds much cooler than AGI.

I have noticed that Claude users seem to be about as intelligent as Claude itself, and wouldn't be able to surpass its output.

I'm giving you, the user, the easiest ability you've most likely ever had to explore embedding space yourself. Embeddings are tricky and can mislead, but they do often compose surprisingly intuitively, especially when you've played and built up a bit of an intuition for it.