Miranda NG Official Community Forum

Forum for English speaking Miranda NG users => Bug reports => Topic started by: Vulpix on 26 08 2018, 10:41:33

Title: Latest libmdbx (commit from 25.8.2018)
Post by: Vulpix on 26 08 2018, 10:41:33
https://github.com/miranda-ng/miranda-ng/commit/c2216a4278bbbc3f breaks miranda's ability to store new messages. That actually also seems to break steam which can no longer log in, guess it depends on something.

I also honestly despise libmdbx at this point, and I wish we could either get the old mmap back (honestly there is nothing good about mdbx), or if you really need something new, migrate to like a verified solution, i.e. sqlite. I understand it'd be slow but right now it's not really fast or at least I can't tell, and the fact the db is 2x bigger, unstable and potentially liable to lose your data with 0 recovery methods is honestly the scariest possible scenario for anyone using an instant messenger aggregator client.

Please reconsider.

That or have a second miranda branch where you do database related changes.

Or at least let me verify the new build before it's pushed out, I don't mind cooperating on this.

EDIT: Tested with @Ghazan and now the change has been rolled back. Should be safe to update again. ( https://github.com/miranda-ng/miranda-ng/commit/cf57ec609dc7b02eb44433514af60bb74951e990 )
Title: Re: Latest libmdbx (commit from 25.8.2018)
Post by: Vulpix on 22 09 2018, 23:29:24
After updating today and losing some data again (thank god for working jsonimport, honestly...), I give up and I don't want to use mdbx ever again :p

https://github.com/miranda-ng/miranda-ng/issues/1597 created, probably going to put a bounty on it.
Title: Re: Latest libmdbx (commit from 25.8.2018)
Post by: Der Jude on 25 09 2018, 23:05:40
Yes please, this MDBX is a joke of a database driver, I'm sick and tired of hearing my HDD constantly scratch because MDBX decides for no good reason to read 100kbyte+write 400kbyte~ worth of I/O operations per second + 1mbyte/s HDD writes for prolonged time, whenever I log in at steam or discord. I never heard my HDD work because of Miranda back when MMAP was being used.

Why can't we just use MMAP as primary database for storing all messages/other stuff MMAP was good for + have secondary database in MDBX for features that cannot be supported by MMAP?
If it wont be possible to make it work with crosslinking references in both DBs then just drop MDBX whole and return to MMAP, MDBX in current form and shape is a big fat mistake.
Title: Re: Latest libmdbx (commit from 25.8.2018)
Post by: Vulpix on 07 10 2018, 16:43:26
Just had another corruption happen today. Not sure what it is honestly. I even remade my profile and reimported it into clean, but seems that didn't matter. It's generally always when I'm talking to someone on discord. Also, when checking past backups, I can generally see that the size of one of the previous backups is randomly different. Like, smaller or larger by 1 megabyte, for example.

Is it possible the backup is doing this? It runs compaction before the actual backup (not a setting I can influence but I believe so). Can this be changed ? Could this be the reason? Maybe the db is so huge the backup takes long enough for some issue to occur during it?

My DB is currently 438MB.

Title: Re: Latest libmdbx (commit from 25.8.2018)
Post by: dartraiden on 08 10 2018, 13:31:37
dbx_sqlite just landed in development version.
Title: Re: Latest libmdbx (commit from 25.8.2018)
Post by: erthink on 26 08 2019, 19:17:33
As the main libmdbx developer I would like to comment:
However, I want to thank both the developers of Miranda-NG (especially @ghazan) and everyone who took part in the testing. In General, this allowed to implement an exclusive mode of working with the database on network drives and significantly advance in testing on the Windows platform.
Title: Re: Latest libmdbx (commit from 25.8.2018)
Post by: ghazan on 27 08 2019, 15:47:32
erthink,
this difference exists because for Miranda libmdbx was synced with master, not with stable/0.2. if you insist that master shall not be used, I can easily roll back libmdbx to 0.2

also, the code of mdbx_load & mdbx_save utilities cannot be imported directly, because of problems of printing cyrillic error messages from the file system. thus using libmdbx as a subtree isn't possible

also, what is the sense of having libmdbx if we coudn't use it as a shared module? in Windows we aren't afraid of using them
Title: Re: Latest libmdbx (commit from 25.8.2018)
Post by: dartraiden on 27 08 2019, 15:55:57
btw, our biggest problem with libmdbx is crashing under Wine
Title: Re: Latest libmdbx (commit from 25.8.2018)
Post by: erthink on 28 08 2019, 10:19:29
ghazan,
Quote
his difference exists because for Miranda libmdbx was synced with master, not with stable/0.2. if you insist that master shall not be used, I can easily roll back libmdbx to 0.2
git subtree and git submodule not only allow this, but are designed to do so.
moreover, there are too many differences to correlate them with the libmdbx's changelog - this is the trouble.

Anyway, I'd like you to put the code in perfect condition.
Including, later, resolving all of my comments on libmdbx' API usage (as far as I remember there were ones).
This is important for durability, performance and minimize database file size.

Quote
also, the code of mdbx_load & mdbx_save utilities cannot be imported directly, because of problems of printing cyrillic error messages from the file system. thus using libmdbx as a subtree isn't possible
Please fill corresponding issue at https://github.com/leo-yuriev/libmdbx/issues (https://github.com/leo-yuriev/libmdbx/issues) and I will fix it.

Quote
also, what is the sense of having libmdbx if we coudn't use it as a shared module? in Windows we aren't afraid of using them
The problem is not libmdbx, but how you collect and use it.
If I am not mistaken, you need support for very old versions of Windows, which had and have problems with Thread-Local-Storage destructors when unloading DLL.
Due this you build libmdbx in the configuration, which can create problems in some scenarios.
However, currently libmdbx inside Miranda is only used by MDBX-driver which is always loaded and unloaded explicitly. Therefore, the embedding libmdbx as a static libraries only in this driver looks safe and rational decision.

dartraiden,
Quote
btw, our biggest problem with libmdbx is crashing under Wine
Likely that the problem in Wine, not in libmdbx.
However, if you fill an issue at https://github.com/leo-yuriev/libmdbx/issues (with backtrace(s), etc) I will try to fix.
For instance - does works or crashes "mdbx_test.exe basic" in case you build the 'clean' mdbx from the original source code?

Title: Re: Latest libmdbx (commit from 25.8.2018)
Post by: ghazan on 28 08 2019, 14:51:29
git subtree and git submodule not only allow this, but are designed to do so.
Submodules work awfully in git... sometimes they aren't updated, sometimes they got broken after branch switching...

However, currently libmdbx inside Miranda is only used by MDBX-driver which is always loaded and unloaded explicitly.
no, Miranda also utilises mdbx_load & mdbx_save as the external compactor, and their code also differs from the standard ones.
Title: Re: Latest libmdbx (commit from 25.8.2018)
Post by: ghazan on 28 08 2019, 15:37:45
erthink,
ok, merged Miranda with 0.3.2