Show HN: A Database Written in Golang

github.com

268 points by Sahil121 2 days ago

Recently created a minimal persistent relational database in Go. Main focus was on implementing & understanding working the of database, storage management & transaction handling. Use of B+ Tree for storage engine(support for indexing), managing a Free List (for reusing nodes), Support for transactions, Concurrent Reads. Still have many things to add & fix like query processing being one of the main & fixing some bugs

Repo link - https://github.com/Sahilb315/AtomixDB

Would love to hear your thoughts

alecco a day ago

Cool!

You watched CMU's intro to Database sytems, right? It's really good and thorough. It will save you of some common pitfalls and can help you navigate the trade-offs.

https://www.youtube.com/watch?v=otE2WvX3XdQ&list=PLSE8ODhjZX...

  • skrtskrt 12 hours ago

    I also recommend "Database Internals" to people. An invaluable book, the logical next read after "Designing Data-Intensive Applications". It's the best and most approachable resource on distributed systems and database consistency / performance tradeoffs I have found.

  • BiraIgnacio a day ago

    Oh thanks for the link, this is really great!

  • Tostino 15 hours ago

    I really have liked the current Optimizer series. So much amazing info.

    I hope some company that heavily relies on Postgres sponsors a project to replace the optimizer with a more modern implementation. I feel like for any large applications with olap style queries (even if on the OLTP database) will benefit incredibly from that work. SQL Servers optimizer was light-years ahead of Postgres when I worked with it last over a decade ago.

    • alecco 9 hours ago

      Sadly, most of those corporations would rather spend millions in proprietary solutions than give back 100k to the community. I've seen it first hand many times. They only give money if they absolutely have to.

  • hobs a day ago

    Looks fun, but the intro requirement is "Knows C++" - that's kinda a non starter right?

dveeden2 a day ago

Other databases written in Go:

- TiDB by PingCAP

- Vitess by PlanetScale

Both are basically only the SQL part as TiDB uses TiKV (written in Rust) and Vitess uses MySQL.

For those who want to implement a database in Go but without having to implement a network protocol there is go-mysql, which allows you to do this: https://github.com/go-mysql-org/go-mysql/blob/master/cmd/go-... As demonstration I created a networked SQLite: https://github.com/dveeden/go-mysql-examples/blob/main/go-my...

Both TiDB and Vitess have parsers that can be used outside standalone. So if you only want to implement your own on disk format, this can help.

Note that I'm working for PingCAP on TiDB and I'm also a co-maintainer for go-mysql.

  • aqueueaqueue 21 hours ago

    We use Tidb at work at scale. Great product! Was looking at the source today to understand an error code.

  • raggi 21 hours ago

    Vitess came from YouTube

    • CBLT 15 hours ago

      Didn't the Vitess team found planetscale?

      • harshitgangal 15 hours ago

        Yes! The founders of PlanetScale were the co-creators of Vitess at YouTube, where it was built to handle MySQL scalability. PlanetScale builds on Vitess but offers a managed, developer-friendly experience.

twalla 2 days ago

Someone asked about resources and I ran into this while evaluating embedded db options for a golang project - it's a collection of db components implemented in golang:

https://github.com/thomasjungblut/go-sstables

  • tjungblut 2 days ago

    thank you for referring, feel free to ask any questions you may have

mbreese 2 days ago

This looks like a good exploratory project!

One thing I’d add to the readme is an example of how you’d use the database in an example application. From the docs, it’s clear that this isn’t a sql database (yet?), so it would be good to have an example to see how to use the database.

It might also be nice to have a description of what happens when you insert or get a record, so others can learn from the code too. Or can you comment here about what your favorite part of the code is? What did you figure out that you didn’t know before? If you’re using the project to learn about databases, what have you learned so far?

  • Sahil121 2 days ago

    thanks man yes currently it is not a fully sql database (have to add query support) My fav part of code is the data retrieval code because had some issues there & got to know lot about it

    Learnt about B+ Trees, transactions, managing concurrent reads & data presistence

voodooEntity 2 days ago

I didnt look into it yet but i already wanted to say - cuz of curiosity i build my own graph database in golang :) and i learned alot. so i absolutly understand why you did it and what experiences you probably made on the way :D

congratz !

  • varelaseb a day ago

    Is this published anywhere?

    • voodooEntity a day ago

      Kinda. So first i went for a standalone application back than to use it as storage/database. This part is archived now. it was released under: https://github.com/voodooEntity/slingshotdb

      At this point i think its important to mention that its nativly an in-memory graph database/storage.

      At some point (4-5 years ago) i decided to change the way i handle it. Since im using the storage in many private projects, and there was no real traction (while also maintaining a database is quite some work) i decided i gonne change the way i handle it away from a standalone database to a library that works as an in-memory graph storage.

      Also i added a custom query language including builder (the queries are 100% json compatible so you can in theory build queries from whatever language and transmit it via json.

      The library is released under : https://github.com/voodooEntity/gits

      Currently im completly reworking the query parser but since this is a privat project and im the only maintainer it may take some to to finish it. The update will takle some smaller bugs in the query parser that can occur on very complex nested queries and also will optimize query speeds. I cant provide a date when the update will occure i only can say it will (because i need it myself).

      Finally said: The strength of gits is to be very fast (hence in memory handling) and easy to use from golang. Its threadsafe by default.

bob1029 2 days ago

If you are wanting proper SQL command support, you could copy the SQLite parser approach. Properly parsing all valid command texts is not a problem that I would find compelling unless I was being compensated for it.

https://www.sqlite.org/lemon.html

You could probably use something like participle, but you'd have to translate the grammar.

https://github.com/alecthomas/participle

pighive 2 days ago

Nice, did you follow a course or any resources to build it? Please share, I have a similar goal. Thank you.

fipar a day ago

Congratulations on a great exploratory project!

It takes me back to my school years. I never got as far as you (not by any long stretch actually), but I did enjoy creating the storage layer of a database from scratch. To actually have to deal with, instead of just think of, all the edge cases, is quite the transformative experience.

As a humble suggestion, since it seems your goal is to understand how relational databases work and not necessarily to write a new database that will compete with others, maybe don't make it an SQL one? We've got enough of those, and not enough of the others. Would be nice to have a new relational DB using Tutorial D as its language for example.

Keep hacking!

  • Sahil121 a day ago

    thanks for the suggestion man, will definitely think about this

mdwhatcott a day ago

Best part: a go.mod with 0 dependencies!

  • OccamsMirror a day ago

    Now that is impressive. Gotta love zero dependency projects.

tapirl a day ago

Cool!

btw, README says Go 1.17+ is required, but go.mod says 1.23+.

eatonphil 2 days ago
whalesalad 2 days ago

How do you connect to it and actually use it? Does it behave like a SQL database? Row-based? Column-based? Good for analytical workloads? Document store? Redis/memcached database? What are it's strengths or weaknesses?

Cool accomplishment in and of itself but hard for anyone here to really give you any criticism or feedback without understanding where it excels and how to work with it.

  • maxmcd 2 days ago

    It does not seem to speak SQL:

        $ git clone git@github.com:Sahilb315/AtomixDB.git
        Cloning into 'AtomixDB'...
        $ cd AtomixDB/
        $ go run .
        Welcome to AtomixDB
        Available Commands:
          CREATE       - Create a new table
          INSERT       - Add a record to a table
          DELETE       - Delete a record from a table
          GET          - Retrieve a record from a table
          UPDATE       - Update a record in a table
          BEGIN        - Begin new transaction
          COMMIT       - Commit transaction
          ABORT        - Rollback transaction
          EXIT         - Exit the program
        
        > create
        Enter table name: foo
        Enter column names (comma-separated): bar,baz
        Enter column types (comma-separated as numbers): 1,2
        Enter indexes (format: col1+col2,col3, ... or leave empty):
        Table 'foo' created successfully.
        > get
        Enter table name: foo
        
        Select query type:
        1. Index lookup (primary/secondary key)
        2. Range query
        3. Column filter
        Enter choice (1, 2 or 3):
    
    
    It looks like records are stored in rows: https://github.com/Sahilb315/AtomixDB/blob/64c95afa8e574595c...

    I do find the source to be well organized and quite readable. Especially if you runs the commands in the cli and then trace how they are each implemented.

    • Sahil121 2 days ago

      yes man it is currently a sql db, have to work on adding query language thanks for checking it out

  • zsoltkacsandi 2 days ago

    > Main focus was on implementing & understanding working the of database

    I think this clearly describes what was the goal.

    • whalesalad 2 days ago

      Yep... but there are dozens of different types of databases out there. There is no way to look at this codebase and give any kind of feedback one way or the other without that understanding, particularly with zero usage examples. Which is what the OP is asking about.

      • jasonlotito 2 days ago

        > There is no way to look at this codebase and give any kind of feedback

        Less than a minute and I know how to use it. It's not complicated. The source code is available and fairly easy. If you can't figure out how to use it in a trivial amount of time, you aren't going to be able to offer anything of value. When did HN go from being about interesting stuff to making bold, ignorant statements like "There is no way to look at this codebase and give any kind of feedback one way or the other without that understanding."

        You should know better.

        • whalesalad 2 days ago

          [flagged]

          • ge96 2 days ago

            yeah as a dev/non-go person don't know how it works either. I assume main.go is where it starts but then what... I see a commands maybe it works by CLI? idk either

            • Xarodon 2 days ago

              Yes it's a CLI REPL, you build and run the binary using the instructions in the README and then create/insert/select from it by typing those commands into the CLI.

              Currently there's no way to connect to it like you would a normal SQL db, but you could embed it like you would sqlite.

              • ge96 2 days ago

                Hopefully there is a help/man

            • arp242 a day ago

              If you're not a developer then you're not the target audience for this.

            • otteromkram 2 days ago

              You're a "dev," but don't seek out test folders/files for use case examples?

              Don't let your employer know...

              • lelandbatey 2 days ago

                They can be a dev and not want to have to clone and explore the code. There's tons of little bits of friction that're not addressed by the "face" of the repo. E.g. The readme doesn't seem to imply how to connect to this database (the answer is you write to its stdin, it's not socket based).

                We're not at work here. Low effort posts are allowed though not encouraged. Implying that making a low effort post somehow makes you not a developer ("you're a 'dev' but don't seem out tests?") is it's own low effort post, while also being hurtful.

                Please, let's keep things kind, focused, and clear.

              • ge96 2 days ago

                [flagged]

  • Sahil121 2 days ago

    it is kind of a sql db but currently does not have the query lang (sql lang) & data is stored in tables similar to other relational DBs. Still have a lot of stuff to add & fix Will add more detail to it

esafak 2 days ago

[flagged]

  • anonzzzies 2 days ago

    "Main focus was on implementing & understanding working the of database, storage management & transaction handling."

    Obviously OP wrote it to understand database theory & implementation.

  • pjmlp 2 days ago

    That a database written in an AOT compiled managed language is possible, for example.

    Specially relevant, because it used to be that writing database engines used to be considered systems programming, and we all know managed languages cannot be possible used for such tasks. /s

  • ajsnigrutin 2 days ago

    It's hackernews, basically every old thing has to be reinvented in the "language of the week", be it go, rust, ruby, or whatever was before that.

    From a learning perspective, nice project for OP, for 'advertising' it, i'd prefer the "what's better than the alternatives" instead of focusing on the language.

    • johnisgood a day ago

      I want more Ada / SPARK and Factor. :D