I’ve always had a hard time understanding the value proposition in the same way I don’t understand the value proposition of e.g. AWS Rekognition.
Paying per use certainly doesn’t make sense, because it has to be qualified by the accuracy you get per use.
And there’s no serious way to understand the accuracy you get per use (on your specific unusual distribution of queries) without employing the expensive ML / stats engineers you probably thought you could avoid hiring by outsourcing to Algolia / Rekognition in the first place.
But once you need to hire them anyway, you might as well utilize them to build this type of thing in-house in ways that are much more tailored and optimized around your in-house data models and data integration tools.
To put in perspective, I’ve worked in several companies (from small start-ups to large ecommerce sites) that have a variety of search needs spanning plug and play Lucene all the way to highly customized joint embedding neural network based nearest neighbor search, and tons in between.
The distribution of text in e.g. the support center search use case was totally different than the product search use case or the document store use case, where highly unique word distribution, special words, frequency of required updates to the search index, asymmetric costs of surfacing bad or deleted content items, etc., was the norm.
Every search use case was different and needed care to develop unique annotated result sets to measure mean reciprocal rank, NDCG, etc., as well as simple stakeholder subjective opinion of quality.
Short of basically hiring Algolia to be a gigantic consultant on all these things, I don’t see how it could actually be valuable.
I suspect it’s just an easy sell to CTO types that don’t really understand. They want “search” to be one problem with one little component to drop in to solve it, but it’s just not real.
You are absolutely factually correct in your analysis, but you are completely missing how business works.
Fundamentally, there is value to most businesses in being able to just buy a decent solution to a non core competency.
That’s where Algolia and AWS and basically all service companies come in... a medium scale clothing manufacturer with a booming e-commerce site may well know they have no clue how to do search, and no clue how to assess and hire individuals who could implement it, and no clue how to find and hire a cio who could put together a team from scratch who could do this on a reasonable timeline.
In my consulting I switched from rolling out ElasticSearch on Azure, AWS or on-prem to Algolia and couldn't be happier. I want to scope and build products and not be a sysadmin - clients don't want to do any of it, let alone hiring fulltime sysadmins.
I have one client in particular that is a stark indicator of this trend - 50+ year old company and their second floor where they used to have 30+ developers and sysadmins and a server room downstairs has now been remodelled into a break room and new offices for their new team of 5 (all awesome replacing a ton of mediocre people who didn't get much done for a decade)
They're doing better, their products are more popular, they don't have to worry about recruiting developers + sysadmins, their current IT staff get paid better and they're saving money.
I find Algolia interesting in that they've managed to capture something that Elasitc didn't - and it could be because of a prevailing wisdom similar to that of grandfather's comment
> Fundamentally, there is value to most businesses in being able to just buy a decent solution to a non core competency.
Bingo. Exactly that. My core competency isn’t (nor do I want it to be,) implementing and maintaining a search system. Same reason I use Twillio, SendGrid, and Heroku.
I’m saying in my experience there is no such thing as one singular “decent solution” for search. It varies enormously from use case to use case, customer cohort to customer cohort, etc.
To even know if you’re buying a decent solution from Algolia or not, you’d already have to hire pretty much all the same staff you’d have to hire to more cost-effectively build it in-house.
I think the fundamental myth, just like with Rekognition, is that if you ship off your data and the third party trains some model (most likely fine-tuning a base model), then you’re done, problem solved.
Even for businesses where search is not a core part of their direct value proposition to customers this is flagrantly untrue.
I am not trying to be flippant, but maybe you are misunderstanding what Algoia provides?
Let's say I have an eCommerce platform. The search provided by the framework is slow and I want to put together an instant search feature. It's too slow, and I don't have a search specialist to speed it up. So I add the Algoia plugin to the platform, sync my products and the work is done. Literally, real world example, install the plugin and suddenly my archaic eCom platform has instant search. Not only that, but I can manage the search result weigths and preferences from within Algolia with no special experience. My existing search couldn't do that.
I am not sure where you would need an ML specialist in any of this, certainly not a whole team. For most people Algolia out of the box is plenty.
> Not only that, but I can manage the search result weigths and preferences from within Algolia with no special experience.
This was a major selling point for us when we switched to Algolia. The business people want to be able to manage things like search without having to go through the programmers, just like they manage GA/GTM and such.
>To even know if you’re buying a decent solution from Algolia or not, you’d already have to hire pretty much all the same staff you’d have to hire to more cost-effectively build it in-house.
You just have to pay Algolia, wire up the APIs, and then see if whatever stakeholder that was complaining about search stops complaining. If they do then it's good enough.
This is correct. Lets say you have an ecom site with product search. How do you know if the search is good? You search for stuff and compare to what you expect.
Even if it sucks, as long as their is revenue lift, you will keep it until the next solution comes along.
And Algolia has great analytics, so you can actually mesure business value from actual search queries: you can tie a query to a purchase and then run further analysis on that. It’s powerful and doesn’t require any exceptional engineering talent to use.
Totally false. This is like massively overfitting a high-order polynomial regression to your data. The fit looks good enough, then the next data point comes in and breaks in a way the existing model cannot be hacked to account for.
The search results you believed were implicitly tuned to some feedback mechanism slowly experience creep as the customer cohort changes and data distribution changes until before you knew it your management of the search solution is a ceaseless game of whack-a-mole siphoning off engineering resources at a rapidly increasing rate. It’s the same false promise of just having some engineers stand up Elastic Search.
Not all decisions are good decisions, it depends on who makes the call. In most cases, someone ask you to use a solution because it looks/feels better. In this case Algolia showed how fast and how well it could be implemented. Once the person who takes the decision is convinced it will be implemented. It's mostly marketing. Probably less than 1% of all e-commerce websites measures the impact of a decision.
I did say as much in my original post: Algolia and Rekognition are marketed at CTOs and directors of engineering who want to be sold on a magical line item that removes a whole concept area from their concern, especially one associated with the difficulty of hiring and affording good machine learning staff who can work on the problem both pragmatically and theoretically. They want to be sold a story.
I will say though that your 1% claim is way off in my experience (which includes 3 medium and large ecommerce companies). These companies employ armies of product managers and analytics staff that measure the shit out of everything from the color of a button to the size of font in a banner display for a discount promo code. These things aren’t usually measured because they find value, rather just to give the appearance of data driven decision making and justify job perpetuity.
You’re not even addressing the engineering costs though. The portion of cost of a search engine solution attributable to the set up of Elastic Search is basically zero. The cost is understanding if the search surfaces relevant items for the specific use case, including asymmetric costs for surfacing bad items in many use cases. Not to mention that plug and play third party solutions like Solr / ES are highly inapplicable to a lot of use cases.
I set up algolia for a client while doing some work for them. We needed to add search to a website we were building which searched over data in the client's CMS.
I could have set up elasticsearch. But algolia was cheap and easy to configure, and we could just click around the algolia UI to tweak things like "number of allowed spelling mistakes". We didn't need to proxy anything or set up routes - we just pulled in the algolia JS library to run queries from the application. It was easier for the client to maintain in an ongoing way rather than maintaining their own elasticsearch instance on EC2 or something like that.
I'm sure there are plenty of times you'd want to run your own elasticsearch instance, and I think that would also have been a reasonable choice for us. But I still feel pretty happy with the choice to use algolia.
Arguments to set up your own elasticsearch instance remind me of the criticism against dropbox - "just run your own server with rsync! Its so easy!". Paying someone a small amount of money to do that for me is often a great deal.
> You’re not even addressing the engineering costs though
This is a pretty silly thing to say. How could you possibly know what I'm addressing?
I actually have experience with all of these scenarios, and bar none Algolia has been the fastest, smoothest, bug-free-and-feature-rich-for-the-dollar rollouts for search experiences I have ever come across.
> including asymmetric costs for surfacing bad items in many use cases
Sure, if you're Google, Netflix or Amazon. For the 99.9% of the rest of the world where search isn't core to the business, there's unlikely to be any discernible impact going one way or the other, except saving money and launching faster.
A better anology would be to consider a person with very special dietary needs, and then say “why buy dinner when you can make it yourself and actually be sure it meets the dietary needs.”
Sounds like a good case for Algolia Professional Services! (If this exists?)
I remember pushing Google's search appliance for a large media company some 10 years ago or so (no benefit to myself though, which was pretty noob). It made sense at the time, and solved something for them better than they probably would have implemented it themselves in a good enough way. The most complicated part was setting up rules about what was public / private / internal etc.
So you are saying it’s too expensive to hire those specialists but you need those specialists to setup a working search solution? Is your stance that only companies that can afford a bespoke solution should implement search?
You do know that paying for these kinds of services from consultants is much more expensive than hiring in-house, right? Consulting is purchased either because you only need the specialization for a short time and can pay the mark up for the flexibility, or because you need some external virtue signalling of prestige or authority to overcome in-house political blockers. Consulting is absolutely not the cost-effective option for a specialization you’ll need frequently.
> you’d already have to hire pretty much all the same staff
I don't know what most Algolia customers are like, but we have been using them for a long time and we are a team of... 2 (only 1 being a technical person).
So hiring a whole team to work on search is certainly not an option for everyone. By paying $29 / month and spending a couple of hours on integration, you get a working search that most small shops will be happy with. That sounds like a valuable proposition to me.
It seems like companies don't have to evaluate it your way? They could informally try it out on a few searches and say, "seems to work better than the other one." This is how Google became popular.
Or maybe do an A/B test. It's a little more formal but doesn't require any search-specific knowledge either.
This approach typically fails quite bad in practice. The subjective impression of success depends on the people around at the time of the decision and the set of queries they chose to inspect. As corner cases pop up with significant cost in production (e.g. surfacing nudity in an image search for a query where it’s highly inappropriate) you become less and less capable of understanding why or hacking business logic in to an already brittle system.
The same problem appears for A/B testing different result orderings. It all hinges on the metrics you chose. If you only looked at e.g. top 5 click through rate, and then later a page redesign brings the top 10 results above the fold, and 6-10 are garbage, you’re suddenly screwed with no systematic way to adjust underlying parameters of the search model that control these things, or really even to get analytics data about it because you felt you could just outsource to a place like Algolia and not have in-house expertise, or that some engineers without statistics backgrounds could just hack it.
Disclaimer: I'm a software engineer working on the core search engine at algolia, but the opinions of this post are my own and not an official statement.
Search is a hard job, and it's hard in many ways. The most obvious difficulties are related to relevance, and yes this part is specific to each business case. But that's not the only issues one has to solve when implementing search.
Even without speaking about the software you run, running it so that you have high availability, fast search results, fast enough to provide search as you type, reliable indexing, low latency in several regions ... This is the first service we provided, this is what SaaS is about. Being on inside of a SaaS compagnie, shows you the amount of works we save to our customers.
Then, about software solution itself. Providing search is not just about running generic piece of code. It's a whole eco system, continuously evolving. Working with a SaaS solution is hiring a team of more than hundred engineers dedicated to search. From the core software to frontend UI modules, the amount of engineering needed is way above what most companies can dedicate to search.
Back to relevance, some aspects are specific to business logic but some are also specific to search. We provide the search knowledge, so that you can focus on your own issues.
And for the software behind our services, we're not trying to build the one size fits all search tool, but a tool dedicated to the kind of search needed in today's web applications. I'm obviously biased, but I strongly believe that the kind of search we focus on, fast accurate top results rather than exhaustive search, fits terribly well our users' needs.
I also do computer vision and search in the areas targeted by Algolia and Rekognition and have not found this to be true at all. For face detection for example, Rekognition was completely unusable for my company.
I haven't used face detection by Rekognition so I can't comment.
However, I'm surprised it's that bad.
I've done two projects in the last 6 months that required face detection, and in both cases combinations of DLib and OpenCV performed perfectly well. Since these are entirely off-the-self models I don't see why Rekognition should - in principle - be any worse.
Yeah, pre-built dlib and opencv models are similarly not realistic for real world applications. We ended up needing to train our own version of MTCNN and separately train celebrity face recognition.
Especially when detecting in images with many faces, these legacy off the shelf things built on Viola-Jones type models or HoG feature extractors are just not acceptable by comparison with deep learning models.
And even at that, you need to fine tune the model to your own specific dataset with appropriate weights to reflect asymmetry in false positives vs false negatives. Simply using any off the shelf model, even a deep CNN model, virtually never works in practice. Unless your real life task is well approximated by the academic data set used for training (and it never is), you’re going to need a computer vision engineer involved.
You mentioned face detection earlier and now you are talking about face recognition. There's a huge difference.
For face detection DLib and OpenCV work really work in real world applications. As I mentioned I've deployed two real-world solutions using them in the past 6 months.
> I’ve always had a hard time understanding the value proposition in the same way I don’t understand the value proposition of e.g. AWS Rekognition.
I signed up for AWS specifically to use Rekognition. I use it to screen alerts from my security cameras. In short, Blue Iris detects motion, a Node-RED flow grabs an image and uses Rekognition to see what's in it, if there's a person detected the Node-RED flow notifies me via PushOver. This significantly reduces the false-positives that inevitably happen on windy days - I've already done a lot of work in Blue Iris on this, but passing alerts through Rekognition makes it almost perfect. Based on my testing, this reduces false-positives to zero and hasn't yet produced a false-negative.
Based on my usage I expect my costs to be ~$5/mo once I'm no longer in the free tier. This is cheaper than the person detection service that Blue Iris natively integrates with and is significantly less effort to get up and running compared to, for example, TensorFlow. I also assume that Amazon will periodically update their detection models to make it better, which is one less thing for me to worry about.
For me, all of these benefits are worth the ~$5/mo.
> “Based on my testing, this reduces false-positives to zero and hasn't yet produced a false-negative.”
But you’re just proving my point. It wouldn’t make sense to use Rekognition unless you had someone with skills to assess the classifier accuracy in the context of your specific problem. For example, it seems like your loss function places an asymmetrically higher cost on false negatives. (Incidentally, it’s interesting you claim it hasn’t produced a false negative ... did you watch every frame of video and make sure?)
If you replace your simple one man operation with a simple loss function on an amount of data you can manually evaluate with instead a complex computer vision workflow, say where face or person detection has legal consequences for a company that sells or licenses stock photography, or an image or video search tool trying to avoid surfacing porn or pirated content, etc. then Rekognition becomes no longer useful, because you’ll need not just one person doing cursory evaluation of false negatives, but a team of people building out a benchmark-like battery of automated evaluations with probably IoU metrics in addition to classifier metrics and will need to figure out how many errors they can tolerate in some cost budget combined with the normal cost budget of usage to Rekognition.
Basically, for some tiny hobbyist use case, I guess it’s fine (though really you could literally just load some Keras model pre-trained on imagenet or some off the shelf version of yolo and save yourself $5/mo) but the value proposition falls apart as soon as the cost function becomes a complicated business one.
Another big difference here is that Algolia does not use machine learning in its algorithms.
This according to an old friend who worked there allowed them to really drill down to why which search results are shown and hence the pay per use does actually make sense.
This sounds like comedy to me. Either people say machine learning is magic and solves everything or people say not to use machine learning at all and this lets them drill down and understand?
Many search tasks really do need machine learning, especially variations on collaborative filter and matrix factorization. Mixed modality search often truly does need deep learning and wasn’t even really possible at a level of fidelity suitable for real use cases until maybe 10 years ago.
If Algolia was categorically omitting a whole class of possible solutions, that would be a big red flag, certainly not a reason to think they can drill down to understand search results better.
I worked once on a large ecommerce search engine that had been built with Solr, and the sort order involved crazy hand-tuned boosting scores applied to ngrams of different sizes. None of it was reproducible, nobody knew where the magic boost weights came from, and as the quality of results started to plummet, there was no way to fix it. Everyone was too afraid to modify the magic constants because even slight perturbations created stark visual errors. And this was just for a super simple non-normalized term frequency matrix with boosts. “Not using machine learning” is not at all a signal that your solution won’t end up as a black box with no interpretability.
> And there’s no serious way to understand the accuracy you get per use (on your specific unusual distribution of queries) without employing the expensive ML / stats engineers you probably thought you could avoid hiring by outsourcing to Algolia / Rekognition in the first place.
You may simply not be able to do this at all. You might not know how to tell good ML/Stats people from bad. You might not be able to pay them competitively.
You might simply want something that's better than nothing, with "nothing" being your realistic alternative. "Expert in-house ML team" is not an alternative many companies can get, and even for the ones that could, it'll take a while. What are you going to do in the meantime?
If you are building a core product, such as Spotify's Discover Weekly. Then no question, you need a dedicated team, and substantial commitment. Maybe even a PhD or two.
But as an unabashed Angolia devotee, I think the value prop of InstantSearch is a no brainer. It's worthwhile looking at the product itself, as an almost textbook example of how to package services to enterprise customers.
It's interesting to see this valuation on search as a service, given the leader, Elastic, is in obvious valuation trouble. The market is unimpressed: the Elastic stock hasn't net moved higher since mid January (so most of the time they've been public).
Just take a look at the actual business performance of Elastic.
$271m in sales for the last fiscal year. Negative $101 million operating loss. Pretty bad, although not extremely unusual for SaaS companies in high growth mode. So there must be great growth going on, right? No.
They added a mere $9m in sales last quarter. $89m in sales with a $42m operating loss (whoa). They added an additional $10m in operating loss and gained a mere $9m in sales.
So if they can keep up that rate of growth, they might generously add $45-$50 million in sales this year. Maybe 16%-18% growth for a company bleeding red ink, that isn't particularly large in terms of sales yet (ie they're struggling to generate fast growth at a small'ish scale). And all that needs to support trading for 20 times sales on a business that is a decade old and has never produced a profit.
Either they find a lot of growth soon or in the next market down cycle Elastic is worth 1/3 to 1/2 of what they're trading for now. The same will probably go for their lesser peers. The clock is starting to tick hard on these extreme valuations (hello WeWork, Uber, Lyft).
I think that Elastic is running into a different issue, with AWS (and possibly other cloud providers) making it hard for them to follow the obvious monetization path (hosting).
Algolia, because it doesn't have an open source offering, doesn't have that issue.
Algolia is also a great project that is easy to implement. I have been a paying customer for years. I have used open source Elastic as well, but it takes a lot more work to run and maintain. I don’t want to spend my time dealing with search, I just want it to work with minimal work. The JS client for Algolia makes it trivial to use — not much work “integrating” it as I have experience with ES.
elastic isn’t competing with algolia, imo. elastic is after the enterprise customers who have a mission-critical need for search (rather than a casual need like algolia users)
but I think mission-critical search is a niche technical problem that isn’t applicable to many enterprises.
Elastic is not even after the search market, but the analytics/loggin/reporting sector. Of course, they are after every customer, but one Elastic rep told me that 75% of customers are not using Elasticsearch for traditional search.
Correct. Elastic competes more with Splunk or Sumologic than it does with Algolia or open-source Lucene/Solr. It's nominally a traditional ecommerce/enterprise search engine as well, but it's not terribly good at it.
Then they are in huge trouble because Kibana is hot garbage. Nearly zero of the user experience goodness from Kibana3 has survived the Elastic aqui-hire.
If they plan to compete with Splunk they need do some serious rework of Kibana. Even something relatively new like Humio is must easier to work with than Kibana.
Entire ELK stack was always pretty bad, compared to Splunk or a syslog server and grep, in my opinion.
We tried to buy a support contract from Elastic years back, but the pricing model put all useful levels of support way beyond our reach. Apparently they didn't understand that we had to make our money and didn't have VC funding.
I have found their search interface for HN (https://hn.algolia.com) useful. They have a drop-down for selecting how you'd want to sort the results. From how I see it Google is doing more blackbox magic whereas Algolia is being a faithful/predictable search engine.
Probably because Algolia isn't using yet any machine learning, but more old-school (yet still very good) tries-based search.
And it's probably why also Algolia shines as search engine for a website vs searching the internet like Google. The former has a small scope (looking for videogames on twich, polo on lacoste) while the latter must be personalized for the user (the snake python vs the programming language python).
If you look at the results, the first 2 results only matched part of the poster's username and many of the top results here are empty threads which are not that useful since HN is mostly for discussions.
To be clear, I'm not saying it's all bad, just pointing out that there are low hanging fruits which can improve results quite a bit.
Also, Algolia seems misspelled in your query which is why it fails when reordering by date: https://imgur.com/a/pO9fBWr, it works otherwise. Although, it says the date is 1 day old when it's just 2 hours old.
Yes, when sorting by date it's currently configured to disable typo-tolerance. This is to avoid having an old (but approximative, like with typos) result BEFORE the correct ones.
I came to this discussion just to say something similar. Algolia doesn’t seem to be great for HN, and off late I see it not loading search results (maybe something to do with my network, browser and the site) or being quite slow compared to say, Google Search.
The results aren't necessarily better because of the Google knowledge of you, because they are pretty good results even in Private browsing without being logged in.
It just seems to me that HN search does a simple keyword matching and reordering based on points without any ranking (please someone correct if I'm wrong).
So freshness doesn't seem to have any value, but the reality is that, at least in tech, very few content is evergreen. Also, when you try to reorder by date for example, you'll end up with a lot of empty threads which again are not that useful. I'm not sure if there's a way of eliminating empty threads from search results.
Their DocSearch program for open source projects is really great [0]. Check out the Jekyll docs [1] instant results that usually find just what you want. My only complaint is I want a plugin and play solution I can deploy for private internal documentation websites.
My guess would be that someone maintaining the main index isn’t maintaining the sorted indices. With Algolia, you have a duplicated index for each type of sort and direction that you want to support.
Yes, when sorting by date it's currently configured to disable typo-tolerance. This is to avoid having an old (but approximative, with typos) result BEFORE the correct ones.
Ah the old multi-factor sort problem. Fun figuring out how to handle those scenarios in Elastic Search too (or more generally any sorting UI). For example you want traditional Chinese food near you. What is better - traditional Chinese food 14km away or fusion 2km away? Well that depends on the wetware, but you have to guess what the user would want.
I was just looking for a site search solution three days ago for a side project I'm working on. I found Elastic's offering but found the lack of a free tier(no 14-day trail crap) off-putting. I'll give these guys a try. Grats on the raise!
Not an exhaustive list, but important differences off the top of my head:
- End-to-end search:
Algolia's offerings span both front-end (InstantSearch drop-in widgets) and back-end (actual search API). Simple applications can be built without ever talking to Algolia API at all because their widgets do it for you.
Atlas is all back-end - just a DB service with FTS on the side; left to you to integrate front-end.
- Configuration:
Algolia's dashboard GUI is where a lot of the configuration is done. Some configurations are not available at all via APIs. It's relatively simple.
Atlas requires more JSON-type configuration entries, and some knowledge of Lucene internals.
- Text analysis:
Algolia text tokenization pipeline is mostly a black-box but works fine most of the time. It exposes only a few settings like ascii-folding. It's fine for normal dictionary words, but has problems with domain-specific text (for example, people/place names, scientific terms, etc).
Atlas exposes many aspects of Lucene's analysis pipeline, but it does require knowledge of Lucene.
- Multilingual support:
Algolia supports all its features for ~70 languages.
Atlas analysis has to be configured separately for each language.
- Query syntax:
Algolia defaults to simple queries but the API supports a more complex query syntax with boolean operators and such.
Atlas has its own JSON query DSL that's related to Lucene's query syntax capabilities.
- Faceting:
Algolia faceting configuration and API are far simpler than Atlas's DSL.
Congrats to them but I find the product to perform quite poorly on HN. Seems to do some kind of fuzzy matching / weird relevance ranking and not full indexing when all I really want is a full index text search.
HN search should be parameterized by the user, where they've commented and what they've clicked on. The last part is why outsourced search performs poorly. They can measure what you click on in their search results but I doubt they know what you clicked on HN.
Side question: how interesting and flexible is algolia as a replacement of a custom solr setup? I don't like the HN search and never heard that it is in use for larger data sets.
You should give it a try; Algolia has a FREE tier you can play with. You can also watch this 45sec video to get a grasp of it: https://youtu.be/IYY5RM1sBC0
If you've ever tried installing ElasticSearch and then switched to Algolia you'll understand how great of a product it is.
Now if Firebase could only buy them out and add decent search to their suite of products that would be swell. Mind boggling that Firebase - part of Google - still lacks a decent search solution.
The homepage is great from a developer perspective, select your backend on the left, frontend on the right, and you get an idea right away of what a basic implementation looks like. Very clever.
I’ve always had a hard time understanding the value proposition in the same way I don’t understand the value proposition of e.g. AWS Rekognition.
Paying per use certainly doesn’t make sense, because it has to be qualified by the accuracy you get per use.
And there’s no serious way to understand the accuracy you get per use (on your specific unusual distribution of queries) without employing the expensive ML / stats engineers you probably thought you could avoid hiring by outsourcing to Algolia / Rekognition in the first place.
But once you need to hire them anyway, you might as well utilize them to build this type of thing in-house in ways that are much more tailored and optimized around your in-house data models and data integration tools.
To put in perspective, I’ve worked in several companies (from small start-ups to large ecommerce sites) that have a variety of search needs spanning plug and play Lucene all the way to highly customized joint embedding neural network based nearest neighbor search, and tons in between.
The distribution of text in e.g. the support center search use case was totally different than the product search use case or the document store use case, where highly unique word distribution, special words, frequency of required updates to the search index, asymmetric costs of surfacing bad or deleted content items, etc., was the norm.
Every search use case was different and needed care to develop unique annotated result sets to measure mean reciprocal rank, NDCG, etc., as well as simple stakeholder subjective opinion of quality.
Short of basically hiring Algolia to be a gigantic consultant on all these things, I don’t see how it could actually be valuable.
I suspect it’s just an easy sell to CTO types that don’t really understand. They want “search” to be one problem with one little component to drop in to solve it, but it’s just not real.
You are absolutely factually correct in your analysis, but you are completely missing how business works.
Fundamentally, there is value to most businesses in being able to just buy a decent solution to a non core competency.
That’s where Algolia and AWS and basically all service companies come in... a medium scale clothing manufacturer with a booming e-commerce site may well know they have no clue how to do search, and no clue how to assess and hire individuals who could implement it, and no clue how to find and hire a cio who could put together a team from scratch who could do this on a reasonable timeline.
In my consulting I switched from rolling out ElasticSearch on Azure, AWS or on-prem to Algolia and couldn't be happier. I want to scope and build products and not be a sysadmin - clients don't want to do any of it, let alone hiring fulltime sysadmins.
I have one client in particular that is a stark indicator of this trend - 50+ year old company and their second floor where they used to have 30+ developers and sysadmins and a server room downstairs has now been remodelled into a break room and new offices for their new team of 5 (all awesome replacing a ton of mediocre people who didn't get much done for a decade)
They're doing better, their products are more popular, they don't have to worry about recruiting developers + sysadmins, their current IT staff get paid better and they're saving money.
I find Algolia interesting in that they've managed to capture something that Elasitc didn't - and it could be because of a prevailing wisdom similar to that of grandfather's comment
What happens if Angolia goes down the tube?
Why would they?
What happens if AWS goes down the tube?
In both cases you complain, hire some people and replace that bit with something else.
> Fundamentally, there is value to most businesses in being able to just buy a decent solution to a non core competency.
Bingo. Exactly that. My core competency isn’t (nor do I want it to be,) implementing and maintaining a search system. Same reason I use Twillio, SendGrid, and Heroku.
I’m saying in my experience there is no such thing as one singular “decent solution” for search. It varies enormously from use case to use case, customer cohort to customer cohort, etc.
To even know if you’re buying a decent solution from Algolia or not, you’d already have to hire pretty much all the same staff you’d have to hire to more cost-effectively build it in-house.
I think the fundamental myth, just like with Rekognition, is that if you ship off your data and the third party trains some model (most likely fine-tuning a base model), then you’re done, problem solved.
Even for businesses where search is not a core part of their direct value proposition to customers this is flagrantly untrue.
I am not trying to be flippant, but maybe you are misunderstanding what Algoia provides?
Let's say I have an eCommerce platform. The search provided by the framework is slow and I want to put together an instant search feature. It's too slow, and I don't have a search specialist to speed it up. So I add the Algoia plugin to the platform, sync my products and the work is done. Literally, real world example, install the plugin and suddenly my archaic eCom platform has instant search. Not only that, but I can manage the search result weigths and preferences from within Algolia with no special experience. My existing search couldn't do that.
I am not sure where you would need an ML specialist in any of this, certainly not a whole team. For most people Algolia out of the box is plenty.
> Not only that, but I can manage the search result weigths and preferences from within Algolia with no special experience.
This was a major selling point for us when we switched to Algolia. The business people want to be able to manage things like search without having to go through the programmers, just like they manage GA/GTM and such.
>To even know if you’re buying a decent solution from Algolia or not, you’d already have to hire pretty much all the same staff you’d have to hire to more cost-effectively build it in-house.
You just have to pay Algolia, wire up the APIs, and then see if whatever stakeholder that was complaining about search stops complaining. If they do then it's good enough.
This is correct. Lets say you have an ecom site with product search. How do you know if the search is good? You search for stuff and compare to what you expect.
Even if it sucks, as long as their is revenue lift, you will keep it until the next solution comes along.
And Algolia has great analytics, so you can actually mesure business value from actual search queries: you can tie a query to a purchase and then run further analysis on that. It’s powerful and doesn’t require any exceptional engineering talent to use.
Totally false. This is like massively overfitting a high-order polynomial regression to your data. The fit looks good enough, then the next data point comes in and breaks in a way the existing model cannot be hacked to account for.
The search results you believed were implicitly tuned to some feedback mechanism slowly experience creep as the customer cohort changes and data distribution changes until before you knew it your management of the search solution is a ceaseless game of whack-a-mole siphoning off engineering resources at a rapidly increasing rate. It’s the same false promise of just having some engineers stand up Elastic Search.
Not all decisions are good decisions, it depends on who makes the call. In most cases, someone ask you to use a solution because it looks/feels better. In this case Algolia showed how fast and how well it could be implemented. Once the person who takes the decision is convinced it will be implemented. It's mostly marketing. Probably less than 1% of all e-commerce websites measures the impact of a decision.
I did say as much in my original post: Algolia and Rekognition are marketed at CTOs and directors of engineering who want to be sold on a magical line item that removes a whole concept area from their concern, especially one associated with the difficulty of hiring and affording good machine learning staff who can work on the problem both pragmatically and theoretically. They want to be sold a story.
I will say though that your 1% claim is way off in my experience (which includes 3 medium and large ecommerce companies). These companies employ armies of product managers and analytics staff that measure the shit out of everything from the color of a button to the size of font in a banner display for a discount promo code. These things aren’t usually measured because they find value, rather just to give the appearance of data driven decision making and justify job perpetuity.
> you’d already have to hire pretty much all the same staff
This is terribly wrong. It's like saying you need as many people to setup Solr/ElasticSearch as you'd need to build a custom search engine.
It requires considerably fewer people to setup, manage, and optimize Algolia than it does to setup, manage, and optimize ES.
Case in point, Twilio.
You’re not even addressing the engineering costs though. The portion of cost of a search engine solution attributable to the set up of Elastic Search is basically zero. The cost is understanding if the search surfaces relevant items for the specific use case, including asymmetric costs for surfacing bad items in many use cases. Not to mention that plug and play third party solutions like Solr / ES are highly inapplicable to a lot of use cases.
I set up algolia for a client while doing some work for them. We needed to add search to a website we were building which searched over data in the client's CMS.
I could have set up elasticsearch. But algolia was cheap and easy to configure, and we could just click around the algolia UI to tweak things like "number of allowed spelling mistakes". We didn't need to proxy anything or set up routes - we just pulled in the algolia JS library to run queries from the application. It was easier for the client to maintain in an ongoing way rather than maintaining their own elasticsearch instance on EC2 or something like that.
I'm sure there are plenty of times you'd want to run your own elasticsearch instance, and I think that would also have been a reasonable choice for us. But I still feel pretty happy with the choice to use algolia.
Arguments to set up your own elasticsearch instance remind me of the criticism against dropbox - "just run your own server with rsync! Its so easy!". Paying someone a small amount of money to do that for me is often a great deal.
> You’re not even addressing the engineering costs though
This is a pretty silly thing to say. How could you possibly know what I'm addressing?
I actually have experience with all of these scenarios, and bar none Algolia has been the fastest, smoothest, bug-free-and-feature-rich-for-the-dollar rollouts for search experiences I have ever come across.
> including asymmetric costs for surfacing bad items in many use cases
Sure, if you're Google, Netflix or Amazon. For the 99.9% of the rest of the world where search isn't core to the business, there's unlikely to be any discernible impact going one way or the other, except saving money and launching faster.
Why do you buy clothes when you could just make them yourself?
A better anology would be to consider a person with very special dietary needs, and then say “why buy dinner when you can make it yourself and actually be sure it meets the dietary needs.”
Sounds like a good case for Algolia Professional Services! (If this exists?)
I remember pushing Google's search appliance for a large media company some 10 years ago or so (no benefit to myself though, which was pretty noob). It made sense at the time, and solved something for them better than they probably would have implemented it themselves in a good enough way. The most complicated part was setting up rules about what was public / private / internal etc.
Except this would be prohibitively expensive. The labor cost of those specialized employees as consultants would be huge.
So you are saying it’s too expensive to hire those specialists but you need those specialists to setup a working search solution? Is your stance that only companies that can afford a bespoke solution should implement search?
You do know that paying for these kinds of services from consultants is much more expensive than hiring in-house, right? Consulting is purchased either because you only need the specialization for a short time and can pay the mark up for the flexibility, or because you need some external virtue signalling of prestige or authority to overcome in-house political blockers. Consulting is absolutely not the cost-effective option for a specialization you’ll need frequently.
> you’d already have to hire pretty much all the same staff
I don't know what most Algolia customers are like, but we have been using them for a long time and we are a team of... 2 (only 1 being a technical person).
So hiring a whole team to work on search is certainly not an option for everyone. By paying $29 / month and spending a couple of hours on integration, you get a working search that most small shops will be happy with. That sounds like a valuable proposition to me.
It seems like companies don't have to evaluate it your way? They could informally try it out on a few searches and say, "seems to work better than the other one." This is how Google became popular.
Or maybe do an A/B test. It's a little more formal but doesn't require any search-specific knowledge either.
This approach typically fails quite bad in practice. The subjective impression of success depends on the people around at the time of the decision and the set of queries they chose to inspect. As corner cases pop up with significant cost in production (e.g. surfacing nudity in an image search for a query where it’s highly inappropriate) you become less and less capable of understanding why or hacking business logic in to an already brittle system.
The same problem appears for A/B testing different result orderings. It all hinges on the metrics you chose. If you only looked at e.g. top 5 click through rate, and then later a page redesign brings the top 10 results above the fold, and 6-10 are garbage, you’re suddenly screwed with no systematic way to adjust underlying parameters of the search model that control these things, or really even to get analytics data about it because you felt you could just outsource to a place like Algolia and not have in-house expertise, or that some engineers without statistics backgrounds could just hack it.
What do you do when you don’t have the budget for a team of in-house experts and you need something that is good enough and you need it now.
Well, what you don’t do is rush to buy a wrong thing because you’re desperate and it has good marketing.
Disclaimer: I'm a software engineer working on the core search engine at algolia, but the opinions of this post are my own and not an official statement.
Search is a hard job, and it's hard in many ways. The most obvious difficulties are related to relevance, and yes this part is specific to each business case. But that's not the only issues one has to solve when implementing search.
Even without speaking about the software you run, running it so that you have high availability, fast search results, fast enough to provide search as you type, reliable indexing, low latency in several regions ... This is the first service we provided, this is what SaaS is about. Being on inside of a SaaS compagnie, shows you the amount of works we save to our customers.
Then, about software solution itself. Providing search is not just about running generic piece of code. It's a whole eco system, continuously evolving. Working with a SaaS solution is hiring a team of more than hundred engineers dedicated to search. From the core software to frontend UI modules, the amount of engineering needed is way above what most companies can dedicate to search.
Back to relevance, some aspects are specific to business logic but some are also specific to search. We provide the search knowledge, so that you can focus on your own issues.
And for the software behind our services, we're not trying to build the one size fits all search tool, but a tool dedicated to the kind of search needed in today's web applications. I'm obviously biased, but I strongly believe that the kind of search we focus on, fast accurate top results rather than exhaustive search, fits terribly well our users' needs.
I do both search and ML solutions in the area that Rekognition targets.
In both cases they are great 80/20 solutions (actually Algoria is more like a 95/5 solution in most cases).
I also do computer vision and search in the areas targeted by Algolia and Rekognition and have not found this to be true at all. For face detection for example, Rekognition was completely unusable for my company.
I haven't used face detection by Rekognition so I can't comment.
However, I'm surprised it's that bad.
I've done two projects in the last 6 months that required face detection, and in both cases combinations of DLib and OpenCV performed perfectly well. Since these are entirely off-the-self models I don't see why Rekognition should - in principle - be any worse.
Yeah, pre-built dlib and opencv models are similarly not realistic for real world applications. We ended up needing to train our own version of MTCNN and separately train celebrity face recognition.
Especially when detecting in images with many faces, these legacy off the shelf things built on Viola-Jones type models or HoG feature extractors are just not acceptable by comparison with deep learning models.
And even at that, you need to fine tune the model to your own specific dataset with appropriate weights to reflect asymmetry in false positives vs false negatives. Simply using any off the shelf model, even a deep CNN model, virtually never works in practice. Unless your real life task is well approximated by the academic data set used for training (and it never is), you’re going to need a computer vision engineer involved.
You mentioned face detection earlier and now you are talking about face recognition. There's a huge difference.
For face detection DLib and OpenCV work really work in real world applications. As I mentioned I've deployed two real-world solutions using them in the past 6 months.
> I’ve always had a hard time understanding the value proposition in the same way I don’t understand the value proposition of e.g. AWS Rekognition.
I signed up for AWS specifically to use Rekognition. I use it to screen alerts from my security cameras. In short, Blue Iris detects motion, a Node-RED flow grabs an image and uses Rekognition to see what's in it, if there's a person detected the Node-RED flow notifies me via PushOver. This significantly reduces the false-positives that inevitably happen on windy days - I've already done a lot of work in Blue Iris on this, but passing alerts through Rekognition makes it almost perfect. Based on my testing, this reduces false-positives to zero and hasn't yet produced a false-negative.
Based on my usage I expect my costs to be ~$5/mo once I'm no longer in the free tier. This is cheaper than the person detection service that Blue Iris natively integrates with and is significantly less effort to get up and running compared to, for example, TensorFlow. I also assume that Amazon will periodically update their detection models to make it better, which is one less thing for me to worry about.
For me, all of these benefits are worth the ~$5/mo.
> “Based on my testing, this reduces false-positives to zero and hasn't yet produced a false-negative.”
But you’re just proving my point. It wouldn’t make sense to use Rekognition unless you had someone with skills to assess the classifier accuracy in the context of your specific problem. For example, it seems like your loss function places an asymmetrically higher cost on false negatives. (Incidentally, it’s interesting you claim it hasn’t produced a false negative ... did you watch every frame of video and make sure?)
If you replace your simple one man operation with a simple loss function on an amount of data you can manually evaluate with instead a complex computer vision workflow, say where face or person detection has legal consequences for a company that sells or licenses stock photography, or an image or video search tool trying to avoid surfacing porn or pirated content, etc. then Rekognition becomes no longer useful, because you’ll need not just one person doing cursory evaluation of false negatives, but a team of people building out a benchmark-like battery of automated evaluations with probably IoU metrics in addition to classifier metrics and will need to figure out how many errors they can tolerate in some cost budget combined with the normal cost budget of usage to Rekognition.
Basically, for some tiny hobbyist use case, I guess it’s fine (though really you could literally just load some Keras model pre-trained on imagenet or some off the shelf version of yolo and save yourself $5/mo) but the value proposition falls apart as soon as the cost function becomes a complicated business one.
Another big difference here is that Algolia does not use machine learning in its algorithms.
This according to an old friend who worked there allowed them to really drill down to why which search results are shown and hence the pay per use does actually make sense.
This sounds like comedy to me. Either people say machine learning is magic and solves everything or people say not to use machine learning at all and this lets them drill down and understand?
Many search tasks really do need machine learning, especially variations on collaborative filter and matrix factorization. Mixed modality search often truly does need deep learning and wasn’t even really possible at a level of fidelity suitable for real use cases until maybe 10 years ago.
If Algolia was categorically omitting a whole class of possible solutions, that would be a big red flag, certainly not a reason to think they can drill down to understand search results better.
I worked once on a large ecommerce search engine that had been built with Solr, and the sort order involved crazy hand-tuned boosting scores applied to ngrams of different sizes. None of it was reproducible, nobody knew where the magic boost weights came from, and as the quality of results started to plummet, there was no way to fix it. Everyone was too afraid to modify the magic constants because even slight perturbations created stark visual errors. And this was just for a super simple non-normalized term frequency matrix with boosts. “Not using machine learning” is not at all a signal that your solution won’t end up as a black box with no interpretability.
> And there’s no serious way to understand the accuracy you get per use (on your specific unusual distribution of queries) without employing the expensive ML / stats engineers you probably thought you could avoid hiring by outsourcing to Algolia / Rekognition in the first place.
You may simply not be able to do this at all. You might not know how to tell good ML/Stats people from bad. You might not be able to pay them competitively.
You might simply want something that's better than nothing, with "nothing" being your realistic alternative. "Expert in-house ML team" is not an alternative many companies can get, and even for the ones that could, it'll take a while. What are you going to do in the meantime?
At our company we have an "Expert in house ML team", but I don't want them wasting there time with search since it isn't our differentiating factor.
If you are building a core product, such as Spotify's Discover Weekly. Then no question, you need a dedicated team, and substantial commitment. Maybe even a PhD or two.
But as an unabashed Angolia devotee, I think the value prop of InstantSearch is a no brainer. It's worthwhile looking at the product itself, as an almost textbook example of how to package services to enterprise customers.
https://www.algolia.com/products/instantsearch/
https://www.algolia.com/enterprise/customers/
It's interesting to see this valuation on search as a service, given the leader, Elastic, is in obvious valuation trouble. The market is unimpressed: the Elastic stock hasn't net moved higher since mid January (so most of the time they've been public).
Just take a look at the actual business performance of Elastic.
$271m in sales for the last fiscal year. Negative $101 million operating loss. Pretty bad, although not extremely unusual for SaaS companies in high growth mode. So there must be great growth going on, right? No.
They added a mere $9m in sales last quarter. $89m in sales with a $42m operating loss (whoa). They added an additional $10m in operating loss and gained a mere $9m in sales.
So if they can keep up that rate of growth, they might generously add $45-$50 million in sales this year. Maybe 16%-18% growth for a company bleeding red ink, that isn't particularly large in terms of sales yet (ie they're struggling to generate fast growth at a small'ish scale). And all that needs to support trading for 20 times sales on a business that is a decade old and has never produced a profit.
Either they find a lot of growth soon or in the next market down cycle Elastic is worth 1/3 to 1/2 of what they're trading for now. The same will probably go for their lesser peers. The clock is starting to tick hard on these extreme valuations (hello WeWork, Uber, Lyft).
I think that Elastic is running into a different issue, with AWS (and possibly other cloud providers) making it hard for them to follow the obvious monetization path (hosting).
Algolia, because it doesn't have an open source offering, doesn't have that issue.
Algolia is also a great project that is easy to implement. I have been a paying customer for years. I have used open source Elastic as well, but it takes a lot more work to run and maintain. I don’t want to spend my time dealing with search, I just want it to work with minimal work. The JS client for Algolia makes it trivial to use — not much work “integrating” it as I have experience with ES.
Maybe Elastic isn't growing because Algolia is eating their lunch?
I guess if one company was to “eat” Elastic real business, this would be Datadog (logs, analytics) not Algolia
elastic isn’t competing with algolia, imo. elastic is after the enterprise customers who have a mission-critical need for search (rather than a casual need like algolia users)
but I think mission-critical search is a niche technical problem that isn’t applicable to many enterprises.
Elastic is not even after the search market, but the analytics/loggin/reporting sector. Of course, they are after every customer, but one Elastic rep told me that 75% of customers are not using Elasticsearch for traditional search.
Correct. Elastic competes more with Splunk or Sumologic than it does with Algolia or open-source Lucene/Solr. It's nominally a traditional ecommerce/enterprise search engine as well, but it's not terribly good at it.
Thus the change of name, dropping "search".
elastic is the parent company that's developping elasticsearch and the associated tools (logstash, kibana, *beats…)
Then they are in huge trouble because Kibana is hot garbage. Nearly zero of the user experience goodness from Kibana3 has survived the Elastic aqui-hire.
If they plan to compete with Splunk they need do some serious rework of Kibana. Even something relatively new like Humio is must easier to work with than Kibana.
Entire ELK stack was always pretty bad, compared to Splunk or a syslog server and grep, in my opinion.
We tried to buy a support contract from Elastic years back, but the pricing model put all useful levels of support way beyond our reach. Apparently they didn't understand that we had to make our money and didn't have VC funding.
Congrats to the team!
Although, am I the only one who finds that searching HN through Google gives better results than searching it through the Algolia powered HN search?
For example, search "ml" in both, Algolia results are years old and don't seem that relevant, whereas Google picks up more recent threads.
I have found their search interface for HN (https://hn.algolia.com) useful. They have a drop-down for selecting how you'd want to sort the results. From how I see it Google is doing more blackbox magic whereas Algolia is being a faithful/predictable search engine.
Probably because Algolia isn't using yet any machine learning, but more old-school (yet still very good) tries-based search.
And it's probably why also Algolia shines as search engine for a website vs searching the internet like Google. The former has a small scope (looking for videogames on twich, polo on lacoste) while the latter must be personalized for the user (the snake python vs the programming language python).
Sample size of 1, but yes, I also tend to have better results using `query site:news.ycombinator.com` than using the built-in Algolia search.
I do this sometimes as well when Angolia isn't giving me the results I'm looking for.
Not really. The sorting is based on popularity by default, which you can change. To compare:
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
https://www.google.com/search?q=news.ycombinator.com%3A+ml&o...
It feels ironic, I searched "Algolia" from both, today's news is ranked as #7 in Algolia powered HN. From Algolia: https://imgur.com/a/atRMnwj From Google: https://imgur.com/a/d3EKdUp
Some people says it's popularity based, but if I change it to date based, it's broken? https://imgur.com/a/HGzL6fO
Simply sorting by date doesn't really seem to be that useful either.
For example, the poster below showed the following link for the "ml" query: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que....
If you look at the results, the first 2 results only matched part of the poster's username and many of the top results here are empty threads which are not that useful since HN is mostly for discussions.
To be clear, I'm not saying it's all bad, just pointing out that there are low hanging fruits which can improve results quite a bit.
Also, Algolia seems misspelled in your query which is why it fails when reordering by date: https://imgur.com/a/pO9fBWr, it works otherwise. Although, it says the date is 1 day old when it's just 2 hours old.
️ oops, a typo in the query, but why they have query correction for popularity based, not date based?
Yes, when sorting by date it's currently configured to disable typo-tolerance. This is to avoid having an old (but approximative, like with typos) result BEFORE the correct ones.
Sort by date works fine. Your last search has a typo `agolia`.
You can change the filters and ranking in the HN search page. I find it faster and easier than Google for comment content.
It's only as good as the HN team have configured the ranking and other algolia settings. Be nice for them to expose these.
Pretty sure HN's search is implemented by Algolia themselves. It's served from algolia.com.
Correct, and the source code is here: https://github.com/algolia/hn-search
You cant self host algolia
No, that wasn't my point. You can't host the service, but you can self-host a site that uses it behind the scenes.
I came to this discussion just to say something similar. Algolia doesn’t seem to be great for HN, and off late I see it not loading search results (maybe something to do with my network, browser and the site) or being quite slow compared to say, Google Search.
Another data point: Have encountered the same problem.
I prefer much more Algolia than Google for searching things in HN.
The customizable stuff are very helpful -- search by user, by type, order stories by popularity or date, etc.
It all depends on what you’re trying to find when you search. Google knows your background, Algolia doesn’t.
I too built a search engine:
https://demo.insideropinion.com/meta_profiles?utf8=%E2%9C%93...
I find it works 100x better for me, when I weight results by my expertise. Basically, returning results based on what I already am familiar with.
The results aren't necessarily better because of the Google knowledge of you, because they are pretty good results even in Private browsing without being logged in.
It just seems to me that HN search does a simple keyword matching and reordering based on points without any ranking (please someone correct if I'm wrong).
So freshness doesn't seem to have any value, but the reality is that, at least in tech, very few content is evergreen. Also, when you try to reorder by date for example, you'll end up with a lot of empty threads which again are not that useful. I'm not sure if there's a way of eliminating empty threads from search results.
FWIW, Private browsing doesn't prevent Google to tailor results to what it can guess about you: https://spreadprivacy.com/google-filter-bubble-study/
They rank by popularity by default (up votes), but I agree generally their API doesn’t seem to do ranking.
You compare a generic solution vs a highly tuned mission critical search engine.
Their DocSearch program for open source projects is really great [0]. Check out the Jekyll docs [1] instant results that usually find just what you want. My only complaint is I want a plugin and play solution I can deploy for private internal documentation websites.
0 - https://community.algolia.com/docsearch/
1 - https://jekyllrb.com/docs/
Congrats to the whole Algolia team.
I can’t think of any SaaS business that invested in, and executed such a smooth onboarding and retention ecosystem.
I’ve used them for small sites and large enterprise clients (10B+) and I’ve always felt like I’ve got way more than I’ve paid for.
PSA: Algolia has basically hidden a category making/taking over strategy in plain sight.
Hacker News generates 86.10% of referral traffic for algolia.com:
https://www.similarweb.com/website/algolia.com#referrals
Amazing product and team. Nicolas still finds time to help small startups like us. Their success is well deserved.
Official blog post: https://blog.algolia.com/algolia-series-c-2019-funding/
Incase someone doesn't know already: they have a HN search too! https://hn.algolia.com/
search for e.g 'microsoft' -> ~120k results
change sort by popularity to sort by date -> ~30k results
I don't really understand how sorting can affect the number of results. Btw youtube search does this too.
My guess would be that someone maintaining the main index isn’t maintaining the sorted indices. With Algolia, you have a duplicated index for each type of sort and direction that you want to support.
Yes, when sorting by date it's currently configured to disable typo-tolerance. This is to avoid having an old (but approximative, with typos) result BEFORE the correct ones.
Ah the old multi-factor sort problem. Fun figuring out how to handle those scenarios in Elastic Search too (or more generally any sorting UI). For example you want traditional Chinese food near you. What is better - traditional Chinese food 14km away or fusion 2km away? Well that depends on the wetware, but you have to guess what the user would want.
What underlying tech does Algolia use? Elastic search?
They built their own. The engine is baked into nginx modules.
https://blog.algolia.com/?filter=engineering
lol
I was just looking for a site search solution three days ago for a side project I'm working on. I found Elastic's offering but found the lack of a free tier(no 14-day trail crap) off-putting. I'll give these guys a try. Grats on the raise!
Does anyone know how MongoDB’s new search engine[0] compares to Algolia?
[0] https://www.mongodb.com/atlas/full-text-search
Not an exhaustive list, but important differences off the top of my head:
- End-to-end search:
Algolia's offerings span both front-end (InstantSearch drop-in widgets) and back-end (actual search API). Simple applications can be built without ever talking to Algolia API at all because their widgets do it for you.
Atlas is all back-end - just a DB service with FTS on the side; left to you to integrate front-end.
- Configuration:
Algolia's dashboard GUI is where a lot of the configuration is done. Some configurations are not available at all via APIs. It's relatively simple.
Atlas requires more JSON-type configuration entries, and some knowledge of Lucene internals.
- Text analysis:
Algolia text tokenization pipeline is mostly a black-box but works fine most of the time. It exposes only a few settings like ascii-folding. It's fine for normal dictionary words, but has problems with domain-specific text (for example, people/place names, scientific terms, etc).
Atlas exposes many aspects of Lucene's analysis pipeline, but it does require knowledge of Lucene.
- Multilingual support:
Algolia supports all its features for ~70 languages.
Atlas analysis has to be configured separately for each language.
- Query syntax:
Algolia defaults to simple queries but the API supports a more complex query syntax with boolean operators and such.
Atlas has its own JSON query DSL that's related to Lucene's query syntax capabilities.
- Faceting:
Algolia faceting configuration and API are far simpler than Atlas's DSL.
Congrats to them but I find the product to perform quite poorly on HN. Seems to do some kind of fuzzy matching / weird relevance ranking and not full indexing when all I really want is a full index text search.
HN search should be parameterized by the user, where they've commented and what they've clicked on. The last part is why outsourced search performs poorly. They can measure what you click on in their search results but I doubt they know what you clicked on HN.
Side question: how interesting and flexible is algolia as a replacement of a custom solr setup? I don't like the HN search and never heard that it is in use for larger data sets.
You should give it a try; Algolia has a FREE tier you can play with. You can also watch this 45sec video to get a grasp of it: https://youtu.be/IYY5RM1sBC0
Are there any good open source alternatives?
If you've ever tried installing ElasticSearch and then switched to Algolia you'll understand how great of a product it is.
Now if Firebase could only buy them out and add decent search to their suite of products that would be swell. Mind boggling that Firebase - part of Google - still lacks a decent search solution.
There is a discussion on Stackshare(a bit old) about the tech they are using:
https://stackshare.io/posts/how-algolia-built-their-realtime...
Long time since those days in a tiny office in Rue du Sentier with Efounders...congrats folks!
I wonder if they can use the money to fix it so it works without Javascript?
Nice work.
The homepage is great from a developer perspective, select your backend on the left, frontend on the right, and you get an idea right away of what a basic implementation looks like. Very clever.
I figure with the ramped up marketing blitz they’ve been on we will be hearing about their IPO soon.