Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Community » Coffee corner » Key/Value store disk-based caching solution (What would fit in well with U++?)
Key/Value store disk-based caching solution [message #52343] Wed, 11 September 2019 07:24 Go to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 220
Registered: June 2011
Experienced Member
I've already looked at lmdb and people have written wrappers for it in C++ but I'd rather have something that fits in better with how U++ works. As far as I can tell it uses a btree and COW with memory mapped files. I think it would be nice, and probably not a lot of work, to write a U++ style wrapper for it unless there are other options. I have an example discord client I'm working on to learn more about U++ and I'd like to be able to cache data like images, Values, etc because it becomes very expensive/slow to grab that data off the Internet each and every time. It would also be nice to have it mmapped so that I don't have to generate copies of that data and can reference it directly.

What would you recommend?
Re: Key/Value store disk-based caching solution [message #52346 is a reply to message #52343] Wed, 11 September 2019 10:20 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
jjacksonRIAB wrote on Wed, 11 September 2019 07:24
I've already looked at lmdb and people have written wrappers for it in C++ but I'd rather have something that fits in better with how U++ works. As far as I can tell it uses a btree and COW with memory mapped files. I think it would be nice, and probably not a lot of work, to write a U++ style wrapper for it unless there are other options. I have an example discord client I'm working on to learn more about U++ and I'd like to be able to cache data like images, Values, etc because it becomes very expensive/slow to grab that data off the Internet each and every time. It would also be nice to have it mmapped so that I don't have to generate copies of that data and can reference it directly.

What would you recommend?


Wrapper would be nice.

Alternatives (from the head): To cache things from the internet (or from over the network), traditional approach to cache them as files works well. Done that in the past.

For key/value storage, if I would be too lazy to use other options (like creating a wrapper), I would use Sqlite3 or MySQL with trivial schema as key/value storage.

Mirek
Re: Key/Value store disk-based caching solution [message #52347 is a reply to message #52346] Wed, 11 September 2019 11:42 Go to previous messageGo to next message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 220
Registered: June 2011
Experienced Member
mirek wrote on Wed, 11 September 2019 10:20

Wrapper would be nice.

Alternatives (from the head): To cache things from the internet (or from over the network), traditional approach to cache them as files works well. Done that in the past.

For key/value storage, if I would be too lazy to use other options (like creating a wrapper), I would use Sqlite3 or MySQL with trivial schema as key/value storage.

Mirek



Yes, right now I'm caching images as files which works just fine, but there is also metadata associated with the images and text that would be very helpful to gather in a KVS. In discord each comment a person makes has a snowflake associated with it and you can start "replaying" missed content while you were offline by providing that snowflake, I believe, at which point it gives you every change made after it (edits, deletes, etc). Right now I'm doing it the "dumb" way and just asking for the last 100 posts but that is not only slow reloading every time you come back online but precludes people from going back even further in the history (pretty much everything in the channel's lifetime is saved in pages).

I think after I've gotten somewhere with it, I'll put it up on github so people can rip through it and tell me how the code can be improved. You've already helped me improve how I'm doing things quite a bit. I'll look into Sqlite3 or see if I can write an lmdb wrapper that has some method calls similar to U++ containers.

[Updated on: Wed, 11 September 2019 11:43]

Report message to a moderator

Re: Key/Value store disk-based caching solution [message #52497 is a reply to message #52343] Thu, 10 October 2019 23:04 Go to previous message
jjacksonRIAB is currently offline  jjacksonRIAB
Messages: 220
Registered: June 2011
Experienced Member
I'm not completely happy with it but I decided I'd release early before I potentially go down the wrong path on this wrapper.

https://github.com/BornTactical/LmdbUpp

It doesn't even support a fraction of the things LMDB does and I'm sure I have bugs so I wouldn't touch anything in production with this, but it does work with Upp serialization. I'll have to iteratively refine it until it becomes something I'm happy with. Nothing is locked down, most everything is subject to change.
Previous Topic: Drawing ellipse with Painter with 2 foci
Next Topic: Future of C++
Goto Forum:
  


Current Time: Fri Apr 19 16:17:17 CEST 2024

Total time taken to generate the page: 0.73755 seconds