Show HN: Shepherd 3.0 – Like wandering the aisles of your favorite bookstore
Hi all, creator here :) - I launched Shepherd.com (https://shepherd.com/) 3 years ago on Hacker News and have added a ton since then! Here is the original Show HN (https://news.ycombinator.com/item?id=26871660).
I’ve interviewed 10,000+ authors & experts to get their 5 favorite reads around different topics, themes, and moods. And I’ve connected those so that you follow your curiosity around topics, authors, books, and more.
Try Art Kleiner’s favorite reads on understanding AI and its effect on people: https://shepherd.com/best-books/understanding-ai-and-its-eff...
Under each book, you can click “What is this book about?” to explore different topics and genres that interest you. I am working on adding themes and other fun connections.
Or you can explore things like...
Places to explore if you like the book Sapiens: https://shepherd.com/search/book/1504
S. B. Divya on her favorite realistic near-future science fiction: https://shepherd.com/best-books/realistic-near-future-scienc...
Places to explore if you like hard sci-fi: https://shepherd.com/search/shelf/12622
Places to explore if you like Stephen King: https://shepherd.com/search/author/4826
Azby Brown’s favorite books on Japanese carpentry and construction: https://shepherd.com/best-books/japanese-carpentry-and-const...
Malayna Evan’s favorite reads on badass women who left a mark on the ancient world: https://shepherd.com/best-books/badass-women-who-left-a-mark...
Shepherd is bootstrapped, and I’ve got many reader features coming soon! I have a newsletter about building the project and early access to new features here: https://forauthors.shepherd.com/newsletter-for-readers
What do we use to build this? Python, Django, Heroku, Postgre, Cloudflare, NLP/ML for Wikipedia topic IDs via Wikifier (https://wikifier.org), Nielsen’s book API database (publisher data + Library of Congress data), and Cloudinary.
My email is ben@shepherd.com if you want to share ideas or suggestions :)
Thanks, Ben
This is cool—I mean it—but what do you mean you interviewed 10,000+ people in three years? How is that possible, given the number of hours in a day? Is it instead that you surveyed 10,000+ people?
I ask because fivebooks.com, which I suppose is a "competitor" of yours, actually does conduct interviews with people about the five books they choose on a subject, and it seems like a lot of work. They only do a couple a week, versus 12-13 a day.
With books, I really don't view anyone as a competitor; the goal should be to help readers find books, as they are the closest thing we have to magic.
Five Books is doing great work, as is https://hardcover.app/ and https://app.thestorygraph.com/. The more book websites we have in the world, the better the world; each one is doing things uniquely.
I am super lucky to have a small part-time team that helps make the publishing process work -> https://forauthors.shepherd.com/team
The publishing process works like this: -> We talk over email to see if they are a good fit and interested -> They fill out a Word doc and return it to us -> Bella & Joy add it to our system and start setting up the books/author profiles -> I screen each list personally for quality and fit (I learned a lot in the first year) -> Our fantastic editors, Frances and Lauren, edit and tweak everything. -> The author gets a preview, and we tweak and adjust as they see fit -> We schedule publishing once they are happy
So, ya, not a video or audio interview, but we do a lot.
Ahh, okay. It was the "I" in "I interviewed" that confused me. I thought it was just you interviewing 13 people every weekday for three years, and maintaining the website by yourself. I think that would either be impossible, or a job fit only for a maniac. I understand now.
I love browsing Shepherd. The more specific lists crack me up but give me so many ideas. I also like that you can browse serendipitously tags related to just about any topic you can search. For example, search time travel -> click "Film", and now you've got a bunch of fun recommendations and you're also thinking about how you could combine those ideas in a new way in your own writing.
https://shepherd.com/bookshelf/time-travel?topics=Q11424
Hoping to be able to combine three ideas even more serendipitously in the future. You sort of can with multiple tags + a topic, but that feels like more of a filter than an additional idea right now.
Thanks! :)
I am working to really improve the accuracy of genres & topics, and adding themes. I am hoping I can tackle that in 3 months or so...
So ... 30 years ago when we were starting Amazon (I was employee #2 there), I had an idea for a feature I wanted in the initial store that is related to what you're trying to do here, but quite different.
While I value human curation, and it probably has to come first, what I wanted to do was to take the LoC classification for a book (typically a triple that has 3 levels of specificity) and build a multidimensional search "space" where you could find books close to the one you were looking at.
In 1994, alas, the LoC data was only available via an insanely slow modem-based download, which made it unusable. The main data source we were using back then (Baker & Taylor / Books in Print) sadly fucked up their record of the LoC classification to that it was impossible to infer where the gaps between the triple's parts were.
The crux of the idea was to leave book classification to librarians, who've been studying this for years and likely doing a much better job than anything us silly software types could come up with.
So to use an example from your list above, in my mental image of the interface I wanted: you'd pick "Sapiens" and see it surrounded by a cloud of other books sharing the same LoC triple (could use other library classification methods too). You could zoom out, and now see even more books sharing only the first two elements of the triple, or even further and end up in "History". Think of it as a "word cloud" made of books, rather than words. You could meander over to another book, and from that book zoom in (and out) of other aspects of the LoC's classification space.
I still wish I had been able to build this, since it mirrored the best part of my experience in large physical bookstores. But it was not to be.
Not sure if the dataset(s) you're using would permit this either, just wanted to mention it.
Yep, I am working on this, and it is a lot of how our system works under the surface. It just isn't accurate enough yet.
I do use the LOC data, but only a very small % of books have that. I also use a NLP/ML tool called Wikifier that connects books to Wikidata topics as I want to use that data to help build knowledge graphs so readers can browse a timeline of WW2 and also see books on those subjects.
What is the problem?
With the topic system, it just isn't quite accurate enough; once I switch over to the newer AI systems later this year, it should work better.
With the genre system, the big problem is that the BISAC publishing genre system is being abused by publishers. So, the data is not reliable because of abuse and ignorance.
For example, I have thousands of books marked nonfiction and fiction because the publisher doesn't know how to use the system and picks nonfiction topics for a fiction book. Or, a publisher picks a book for a genre it doesn't belong in because they think it will get them more sales.
It is really frustrating and I am spending a huge chunk of my time this year to look at how I can fix it. It probably means ignoring this system and using AI to classify it with other data we have.
I remember your first Show HN post. I was immediately hooked with what seems like a very diverse fivebooks alternative. Back then I thought it was like your hobby project and would probably fizzle out within a few months like many other projects, but nope, you just keep interviewing and making new lists days after days. Since then, clicking the shuffle button on your search site has become my favorite pastime :) Thank you for this site.
That is so nice to hear; thanks for gifting me with a huge smile today :)
There have been some tough moments in the last year, especially with funding, but we are on a good path and I am having fun -> https://build.shepherd.com/p/shepherd-stress-path-forward
Super excited to ship reader features soon!
btw, here are some other fun explore points :)
Sci-Fi -> https://shepherd.com/search/shelf/9008
History -> https://shepherd.com/search/shelf/12669
Fantasy -> https://shepherd.com/search/shelf/12664
Behavioral economics -> https://shepherd.com/search/shelf/3105
One HUGE challenge we have right now is that the publishers abuse their genre system, and the NLP/ML system I use for topic identification isn't great. I am working on a new system to bypass both of those and hope to roll that out later this year; that will drastically improve accuracy on topics/genres.
I was surprised to not see Sci-Fi on the front page!
Yep its there; just use search as we have ~3500 genres and topics in the system :)
There is also a page for the best sci-fi books of 2023 that authors picked. We asked authors to share their 3 favorite reads of the year and then tabulated the results for new books and books published at any time -> https://shepherd.com/bboy/2023/science-fiction
When I travel, I love visiting old independent bookstores and wandering around to discover new books. The creakier the floorboards, the better. This is awesome - thank you!
I love indie bookstores :). I "stole" the original idea from a fantastic indie bookstore that I stumbled into.
I walked into the bookstore, and they had little index cards to show off staff favorites and a little half-tweet on why: "Andy loves the characters in this outside-the-box fantasy series;- you will dig it."
It was this magical way to meet the books. I thought it would be fun to try to do that while helping authors meet readers.
Big chains do this too
What is the source for a book description?
I ask because I just started listening to "Straight Man" by Richard Russo and looked it up on Shepherd. Shepherd mentions that "Straight Man" is being made into a TV series. That's out of date and Wikipedia has the up to date info (one season of a series based on "Straight Man" was created and called "Lucky Hank").
Cool site! Thanks for bringing it to my attention.
I immediately found two books I'd not come across before, that I am going to start today. This is a great site, thank you
That is so great to hear! :)
I am working on new bookshelf pages to expand that section. I want to add a more visual format and break it apart a bit more.
Check out the early mockup here: https://www.dropbox.com/scl/fi/mrvkev7pft5luvj10eqfc/Screens...
Why does searching for "mathematics" only provide specific texts? No general category/listing?
Ya, sorry, I have some issues with Algolia weighting; I am working on getting topics/genres to outrank author names as "math" is causing some problems there.
Here is what should show up:
Math bookshelf page - https://shepherd.com/bookshelf/math
Math search results page - https://shepherd.com/search/shelf/693
Perfecting timing, just finished my book.
Loving the site, thankyou
Nice, let me know what you end up with :)
Still a ton of rough spots, so feel free to send me any frustrations (ben@shepherd).
Hi Ben. Great project. I'm playing with it right now in looking for "comps". Will ping with feedback.
Ya, a lot of publishers and publicists mention that :)
As I launch reader reviews, I also want to feed into that; I plan to prompt them with "What book do you love that is like this book?" I think that is great for other readers and it helps give authors a true comp from the eyes of a reader (ETA late this year or early next I hope).
I absolutely love your domain!
Very lucky snag way back in the day :)
I also own pilgrimage.com; I'd love to do something like this for music one day with the tagline, "Life is a journey; let's give it a soundtrack"!
I thought this was about GNU shepherd but it's blog spam
Browsing this website gives me the urge to poop for some reason.
[dead]
Except the selections of what you get to look at are made by an algorithm, not people. Yes, an algorithm that mixes and matches ratings made by (a particular category of) people. But at the end of the day - an algorithm.
So in this very fundamental sense, it's nothing like your local bookstore at all.
Hmm, can you explain?
Every book on our website is chosen by a reader who loved it.
Here is an example: https://shepherd.com/best-books/mystery-and-suspensewith-a-b...
The only books within our website are those that a reader loves and picked as part of their book rec list, or fav 3 reads of the year.
Every book on our website is chosen by a reader who loved it.
Yeah, we get that. It's the aggregate ranking of such picks (taken across a large set of reviewers somehow) that I was referring to. That is, an algorithm.
Which is, again, entirely different from how your local bookstore operates (for any store worth visiting, that is).
No, sorry, that is incorrect.
Without explaining why. And then you walk off.
But okay, I guess this this how you talk to people.
Sorry, I didn't know what to say; I had explained it previously. The only books on our website are those picked by an author. There is no "aggregate" rating system of any kind or even a rating system.
The only "kinda" system we have is for our favorite 3 reads of the year, we showed which books were picked the most.
There is no "aggregate" rating system of any kind or even a rating system.
Yes there is. Pick a genre, and you get a list of books explicitly ranked (rated, same thing) by the number of people who reviewed it (in essence, by "likes").
So not surprisingly you get authors like Stephen King, Michio Kakio, etc floating the top. The same predictable blandness you get at any large chain bookstore.
No, the default sort is serendipity :). You have to actively choose “most recommend “. Can you try again to use a bookshelf?
I was referring to the "Best Books of 2023" section, which (for every sublink) produces the behavior I'm describing.
So okay, fine - only part of the site is based on straight-up favorite counting. But then I did some topical searches, and the result definitely appears to be algorithmic as well (judging by the large number of plainly off-topic matches).
I'm not saying this is a horrible site or that there's no potential for the basic idea behind it. It definitely seems to be more appealing, and potentially more useful than most online bookstores I've seen.
But whatever you're doing - it's definitely not the same as actual human curation.
Yep, that is just a small section with a combined total of authors ' favorite reads of the year :) that is only 1% of the website. Just try search for any book, author, topic, or genre you are interested in.
Here are some explore points you can try via search:
Sci-fi explore point - https://shepherd.com/search/shelf/9008
Sci-fi bookshelf - https://shepherd.com/bookshelf/science-fiction
Book list by authors - https://shepherd.com/best-books/science-fiction-for-fantasy-...
Just try search for any book, author, topic, or genre you are interested in
In my post above: "Then I did some topical searches ..." For large / concrete topics, the result sets look good enough. For smaller / more nebulous topics, they start to fall apart. Which is not the end of the world, of course. But which definitely leaves behind a distinctly non-human aftertaste, shall we say.
Look, I have the feeling we're talking past each other. It's your site, go ahead and have fun with it.
Will do, just wanted to make sure you realized it wasn't that small award feature we did :)