I've been considering open sourcing my search engine. Search is like a fractal of interesting problems, and pretty much every aspect of the search engine has known areas of improvement, so I'm sure it would be a fun project to collaborate on.
I'm honestly a bit at a loss how to actually go about it, since it's not an application or a library where others are expected to run it, but a fairly bespoke piece of web service that requires specific hardware configurations to do anything useful (as well as extremely unwieldy datasets). The only rolemodel I can find is something like Wikipedia.
You have to do a lot of advertising. Not buying ads on Facebook, but talking about the project and trying to convince people to join. Putting a project up on GitHub and then sitting back and waiting for people to show up will result in exactly 0 people showing up. You'll need to identify where your target developers are, currently, and go to them. It helps to get interviewed in tech blogs, too.
I don't think finding developers is my first obstacle, I have a bunch of people who have contacted me and shown interest in participating.
The problem is the logistics of an open search project that is a web service with serious hardware requirements. I think the minimum hardware requirements is about 14 Gb of RAM, that's without any real data loaded into the system. Testing is awkward and cumbersome, and the data logistics are a real headache even on the same network as the production instance.
To even run the search engine, you need a few hundred megabytes of language models, as well as a probably few gigabytes of website data to conduct meaningful testing. The production instance has a disk footprint of about half a terabyte.
I’m not sure how one could enable others to contribute easily but just making the code open source as is would be interesting. Those with a passing interest in search engines can browse the code of a capable engine and pick up some knowledge. While those who really want to help out will find a way to run it and contribute (in the process helping document and maybe making it easier).
My philosophy to open source is just do my personal projects in the open even if it’s not “accessible” and if someone comes along and likes it they can contribute or fork it and modify to their liking. Or they might just browse the code and pick up something in passing.
Are those hard requirements and, if so, why? It might be worthwhile to add a "local" or "dev" mode which loads a very minimal amount of data (maybe only a handful of items are indexed and only a handful of search terms that work with them).
My first question would be: what's your goal? Why do you want to open source it?
Because if the answer is "knowledge sharing" then just document it and open it, that's it, whoever is interested will show up (and if nobody does you don't care, your goal is fulfilled). If instead your goal is to eventually build a community around it then you'll have to put more effort (for example talk in conferences).
You get my point, you will know which road to take as soon as you know where you want to go.
I run an Open Source project[0] and I'm in a group with a bunch of other Open Source founders. I've spent a lot of time thinking about these types of problems and I've put down some of them into a post here[1].
My email is on my profile if you'd like to chat directly. That offer extends to anybody else, too!
So this is a bit tangential.
I've been considering open sourcing my search engine. Search is like a fractal of interesting problems, and pretty much every aspect of the search engine has known areas of improvement, so I'm sure it would be a fun project to collaborate on.
I'm honestly a bit at a loss how to actually go about it, since it's not an application or a library where others are expected to run it, but a fairly bespoke piece of web service that requires specific hardware configurations to do anything useful (as well as extremely unwieldy datasets). The only rolemodel I can find is something like Wikipedia.
I'm curious if anyone knows good "role models"?
Sounds similar to ML research
You have to do a lot of advertising. Not buying ads on Facebook, but talking about the project and trying to convince people to join. Putting a project up on GitHub and then sitting back and waiting for people to show up will result in exactly 0 people showing up. You'll need to identify where your target developers are, currently, and go to them. It helps to get interviewed in tech blogs, too.
I don't think finding developers is my first obstacle, I have a bunch of people who have contacted me and shown interest in participating.
The problem is the logistics of an open search project that is a web service with serious hardware requirements. I think the minimum hardware requirements is about 14 Gb of RAM, that's without any real data loaded into the system. Testing is awkward and cumbersome, and the data logistics are a real headache even on the same network as the production instance.
To even run the search engine, you need a few hundred megabytes of language models, as well as a probably few gigabytes of website data to conduct meaningful testing. The production instance has a disk footprint of about half a terabyte.
Have you asked that bunch of people how they would like to cooperate?
My ignorant suggestion towards sharing those gigabytes of data: have you considered bittorrents?
I’m not sure how one could enable others to contribute easily but just making the code open source as is would be interesting. Those with a passing interest in search engines can browse the code of a capable engine and pick up some knowledge. While those who really want to help out will find a way to run it and contribute (in the process helping document and maybe making it easier).
My philosophy to open source is just do my personal projects in the open even if it’s not “accessible” and if someone comes along and likes it they can contribute or fork it and modify to their liking. Or they might just browse the code and pick up something in passing.
Are those hard requirements and, if so, why? It might be worthwhile to add a "local" or "dev" mode which loads a very minimal amount of data (maybe only a handful of items are indexed and only a handful of search terms that work with them).
My first question would be: what's your goal? Why do you want to open source it?
Because if the answer is "knowledge sharing" then just document it and open it, that's it, whoever is interested will show up (and if nobody does you don't care, your goal is fulfilled). If instead your goal is to eventually build a community around it then you'll have to put more effort (for example talk in conferences).
You get my point, you will know which road to take as soon as you know where you want to go.
My 2c.
I run an Open Source project[0] and I'm in a group with a bunch of other Open Source founders. I've spent a lot of time thinking about these types of problems and I've put down some of them into a post here[1].
My email is on my profile if you'd like to chat directly. That offer extends to anybody else, too!
[0]: https://github.com/lunasec-io/lunasec/
[1]: https://www.lunasec.io/docs/blog/how-to-build-an-open-source...