embedded database options for Nodejs

Asked by 1 year ago
Hi, What are the options for embedded database? I use redis and mongodb for now, but sometimes you made some small apps, and does not want to mix data with existing redis db or mongodb. it should be easier to install, now I'm looking at nosql, https://npmjs.org/package/nosql, also ejdb, https://npmjs.org/package/ejdb, but seems you can not have your own data file for a individual app. sqllite is another, but it's not json based, any suggestions? Thanks, Angelo

Your Answer

Name:
Reply:

All Answers

Answer by 1 year ago
https://github.com/rvagg/node-levelup is one option. Trygve
Answer by 1 year ago
Check out levelDB: https://npmjs.org/package/levelup
Answer by 1 year ago
Consider LevelDB https://github.com/rvagg/node-levelup
Answer by 1 year ago
looks like level db is a good option with all the recommendations, is it easier to install? it seems depends on some C libs.
Answer by 1 year ago
Quoted message by Michael Nisi 1 year ago
Consider LevelDB https://github.com/rvagg/node-levelup
npm install levelup
Answer by 1 year ago
Hey Angelo, I'm working on an a persistent key-value store in Node right now. It's called Medea, and it's currently 100% JavaScript. https://github.com/argo/medea It's pre-1.0 at this time, but I think the API is pretty stable. Julian Gruber maintains a benchmark of databases/libraries called from Node. You can find that here: https://github.com/juliangruber/multilevel-bench Medea (10.000x) 12,324 op/s ⨠ set small 12,313 op/s ⨠ set medium 12,248 op/s ⨠ set large 40,566 op/s ⨠ get large 44,246 op/s ⨠ get medium 45,174 op/s ⨠ get small levelUP (10.000x) 38,374 op/s ⨠ set small 33,019 op/s ⨠ set medium 23,348 op/s ⨠ set large 30,622 op/s ⨠ get large 36,191 op/s ⨠ get medium 38,326 op/s ⨠ get small This is using an earlier version of Medea. The "set" numbers for Medea are a lot better now than they were at that time, though I believe levelUP is still doing faster writes. LevelUP (and LevelDB) have a few differences. In Medea, a range query would have to be built from the calling code. You'd have to filter listKeys() and then iterate over those keys doing a get on each one. Medea keeps a copy of all keys and value file IDs/offsets in-memory (to make a maximum of 1 disk seek per get), so your key set has to fit in memory. It's working pretty good for me, and I know at least one other person who is using it for their project. Medea is pretty stable now, but I expect stability issues to be raised and squelched as we approach 1.0. That said, levelUP and LevelDB are great projects, and you can't go wrong with those options, either. Medea being all-JavaScript is a pro for me at this time. Cheers,
Answer by 1 year ago
https://github.com/developmentseed/node-sqlite3 I've used this quite extensively on linux/windows/mac and even Raspberry PI
Answer by 1 year ago
There's dirty for super simple storage: https://github.com/felixge/node-dirty
Answer by 1 year ago
Hi Ben, this is interesting, I can find good use case for that, the repository is of felixge, but the author is you, does that men felixge is no longer in this project? Thanks, Angelo
Answer by 1 year ago
Hi Kevin, I took a look, the 100% js codebase is one plus factor, one thing, is there any utility to browse the database? Thanks, Angelo
Answer by 1 year ago
Quoted message by Ben Taber 1 year ago
There's dirty for super simple storage: https://github.com/felixge/node-dirty
Felix is still the owner and wrote most of the code, I've just been helping out lately.
Answer by 1 year ago
Quoted message by Kevin Swiber 1 year ago
Hey Angelo, I'm working on an a persistent key-value store in Node right now. It's called Medea, and it's currently 100% JavaScript. https://github.com/argo/medea It's pre-1.0 at this time, but I think the API is pretty stable. Julian Gruber maintains a benchmark of databases/libraries called from Node. You can find that here: https://github.com/juliangruber/multilevel-bench Medea (10.000x) 12,324 op/s ⨠ set small 12,313 op/s ⨠ set medium 12,248 op/s ⨠ set large 40,566 op/s ⨠ get large 44,246 op/s ⨠ get medium 45,174 op/s ⨠ get small levelUP (10.000x) 38,374 op/s ⨠ set small 33,019 op/s ⨠ set medium 23,348 op/s ⨠ set large 30,622 op/s ⨠ get large 36,191 op/s ⨠ get medium 38,326 op/s ⨠ get small This is using an earlier version of Medea. The "set" numbers for Medea are a lot better now than they were at that time, though I believe levelUP is still doing faster writes. LevelUP (and LevelDB) have a few differences. In Medea, a range query would have to be built from the calling code. You'd have to filter listKeys() and then iterate over those keys doing a get on each one. Medea keeps a copy of all keys and value file IDs/offsets in-memory (to make a maximum of 1 disk seek per get), so your key set has to fit in memory. It's working pretty good for me, and I know at least one other person who is using it for their project. Medea is pretty stable now, but I expect stability issues to be raised and squelched as we approach 1.0. That said, levelUP and LevelDB are great projects, and you can't go wrong with those options, either. Medea being all-JavaScript is a pro for me at this time. Cheers,
Angelo, Not at this time. Implementing a browser shouldn't be too difficult, though. Medea stores both keys and values as Buffer objects. You could iterate through the key-value store and stringify key and value Buffers prior to display. There are a number of Web front-end data grids you could use for the UI.
Answer by 1 year ago
Quoted message by Angelo Chen 1 year ago
Hi Kevin, I took a look, the 100% js codebase is one plus factor, one thing, is there any utility to browse the database? Thanks, Angelo
Hi Kevin, just tried, got a question, is there a way to open() at beginning, then use it later? the sample is, you have to put the code in a callback, this does not work: ar Medea = require('medea'); var medea = new Medea(); medea.open() medea.get('hello', function(err, val) { console.log(val.toString()); medea.close(); });
Answer by 1 year ago
Hi Ben, looks like, you have to put access code in db.on('load', ...) how to just open the database once at beginning, then use later? say all the database code will be in mydao.js, exporting some CRUD methods, so I can call from another js, dao.append({id:1, data:'d'}) in this append method, I have to wrap the function in a db.on('load', function(rec){}) ? Angelo
Answer by 1 year ago
Quoted message by Kevin Swiber 1 year ago
Angelo, Not at this time. Implementing a browser shouldn't be too difficult, though. Medea stores both keys and values as Buffer objects. You could iterate through the key-value store and stringify key and value Buffers prior to display. There are a number of Web front-end data grids you could use for the UI.
There's no synchronous open at this time. This has been an issue a couple of times when porting apps to Medea as most Node database libraries use synchronous database initialization by default. Adding db.openSync is more time-consuming work than you may think. There's a lot going on when you open a new Medea database. It's the most expensive operation, because it: 1) checks for a directory (creates if necessary) 2) opens a new data file for writing 3) opens a new hint file for writing 4) creates a lock file 5) writes the pid to the lock file 6) scans existing files 7) loads key to file ID and offset mappings from each file into memory To create a sync version of all that means creating *Sync methods all over the place. I don't think this will be something we introduce in v1. For servers, I usually…. db.open(function() { server.listen(process.env.PORT || 3000); });
Answer by 1 year ago
Quoted message by Ben Taber 1 year ago
There's dirty for super simple storage: https://github.com/felixge/node-dirty
No guarantees that you can read a record before .load has been called, but you can write before load. https://github.com/felixge/node-dirty#dirty-event-load-length To just open the file once on app init, wrap it in its own module, and then require that into other modules. Something like below would work. -- db.js var db = require('dirty')('./data.db'); var isLoaded; db.on('load', function() { isLoaded = true; }); db.ready = function(cb) { if (isLoaded) { return cb(); db.on('load', cb); module.exports = db; -- otherFile.js var db = require('./db.js'); db.ready(function() { db.get(); db.set(); // etc })
Answer by 1 year ago
Quoted message by Angelo Chen 1 year ago
Hi Ben, looks like, you have to put access code in db.on('load', ...) how to just open the database once at beginning, then use later? say all the database code will be in mydao.js, exporting some CRUD methods, so I can call from another js, dao.append({id:1, data:'d'}) in this append method, I have to wrap the function in a db.on('load', function(rec){}) ? Angelo
When using these embedded databases, like node-levelup, how do you handle cluster based apps? I'd love to use an embedded database, but also need to run our express app on all 8 cores of the machine. I know I can process.send to children & master, but that seems like a clunky way of interfacing. Creating a "db" process separate that responds over HTTP and have the workers do requests against that is possible, but not sure if there's a more common / clean way of implementing.
Answer by 1 year ago
Quoted message by Ben Taber 1 year ago
No guarantees that you can read a record before .load has been called, but you can write before load. https://github.com/felixge/node-dirty#dirty-event-load-length To just open the file once on app init, wrap it in its own module, and then require that into other modules. Something like below would work. -- db.js var db = require('dirty')('./data.db'); var isLoaded; db.on('load', function() { isLoaded = true; }); db.ready = function(cb) { if (isLoaded) { return cb(); db.on('load', cb); module.exports = db; -- otherFile.js var db = require('./db.js'); db.ready(function() { db.get(); db.set(); // etc })
Hey John, I created medea-clusterify[1] to handle this use case. It's using worker.send and process.send to do exactly as you describe. This is not ideal, but it works. The master process holds the Medea instance, and the workers communicate with it. This means the master process can bottleneck. However, you can still handle a lot more connections concurrently. Doing a key-space partition is another way to handle this. To do that, you have to route traffic between processes based on owned key-space. There has been some discussion about this in a GitHub issue[2]. As discussed in this issue, cross-platform memory mapping would be a good way to not duplicate in-memory data structures per process, but that won't come until at least v2. LevelDB (and levelUP) restricts usage to a single process, as well, though you could build some multi-process support around it. [1] https://github.com/argo/medea-clusterify [2] https://github.com/argo/medea/issues/10
Answer by 1 year ago
Quoted message by Ben Taber 1 year ago
No guarantees that you can read a record before .load has been called, but you can write before load. https://github.com/felixge/node-dirty#dirty-event-load-length To just open the file once on app init, wrap it in its own module, and then require that into other modules. Something like below would work. -- db.js var db = require('dirty')('./data.db'); var isLoaded; db.on('load', function() { isLoaded = true; }); db.ready = function(cb) { if (isLoaded) { return cb(); db.on('load', cb); module.exports = db; -- otherFile.js var db = require('./db.js'); db.ready(function() { db.get(); db.set(); // etc })
For those interested in leveldb this recent NodeUp is worth a listen: http://nodeup.com/fortyone.
Answer by 11 months ago
Quoted message by John Fitzgerald 1 year ago
When using these embedded databases, like node-levelup, how do you handle cluster based apps? I'd love to use an embedded database, but also need to run our express app on all 8 cores of the machine. I know I can process.send to children & master, but that seems like a clunky way of interfacing. Creating a "db" process separate that responds over HTTP and have the workers do requests against that is possible, but not sure if there's a more common / clean way of implementing.
I'm a bit late to the party but you can try NeDB : https://github.com/louischatriot/nedb It's a pure javascript database with no external dependency, that implements the most common part of MongoDB's API. You can use it as an in-memory only or persistent datastore. Le samedi 20 avril 2013 15:52:58 UTC+2, Gabriel Farrell a écrit :
Answer by 7 months ago
Try unqlite which has a binding for node.js https://npmjs.org/package/unqlite