Author Topic: Latest libmdbx (commit from 25.8.2018)  (Read 7956 times)

0 Members and 1 Guest are viewing this topic.

Offline Vulpix

Latest libmdbx (commit from 25.8.2018)
« on: 26 08 2018, 10:41:33 »
https://github.com/miranda-ng/miranda-ng/commit/c2216a4278bbbc3f breaks miranda's ability to store new messages. That actually also seems to break steam which can no longer log in, guess it depends on something.

I also honestly despise libmdbx at this point, and I wish we could either get the old mmap back (honestly there is nothing good about mdbx), or if you really need something new, migrate to like a verified solution, i.e. sqlite. I understand it'd be slow but right now it's not really fast or at least I can't tell, and the fact the db is 2x bigger, unstable and potentially liable to lose your data with 0 recovery methods is honestly the scariest possible scenario for anyone using an instant messenger aggregator client.

Please reconsider.

That or have a second miranda branch where you do database related changes.

Or at least let me verify the new build before it's pushed out, I don't mind cooperating on this.

EDIT: Tested with @Ghazan and now the change has been rolled back. Should be safe to update again. ( https://github.com/miranda-ng/miranda-ng/commit/cf57ec609dc7b02eb44433514af60bb74951e990 )
« Last Edit: 26 08 2018, 13:14:45 by Vulpix »
 

Offline Vulpix

Re: Latest libmdbx (commit from 25.8.2018)
« Reply #1 on: 22 09 2018, 23:29:24 »
After updating today and losing some data again (thank god for working jsonimport, honestly...), I give up and I don't want to use mdbx ever again :p

https://github.com/miranda-ng/miranda-ng/issues/1597 created, probably going to put a bounty on it.
 

Offline Der Jude

  • Newbie
  • *
  • Posts: 29
Re: Latest libmdbx (commit from 25.8.2018)
« Reply #2 on: 25 09 2018, 23:05:40 »
Yes please, this MDBX is a joke of a database driver, I'm sick and tired of hearing my HDD constantly scratch because MDBX decides for no good reason to read 100kbyte+write 400kbyte~ worth of I/O operations per second + 1mbyte/s HDD writes for prolonged time, whenever I log in at steam or discord. I never heard my HDD work because of Miranda back when MMAP was being used.

Why can't we just use MMAP as primary database for storing all messages/other stuff MMAP was good for + have secondary database in MDBX for features that cannot be supported by MMAP?
If it wont be possible to make it work with crosslinking references in both DBs then just drop MDBX whole and return to MMAP, MDBX in current form and shape is a big fat mistake.
 

Offline Vulpix

Re: Latest libmdbx (commit from 25.8.2018)
« Reply #3 on: 07 10 2018, 16:43:26 »
Just had another corruption happen today. Not sure what it is honestly. I even remade my profile and reimported it into clean, but seems that didn't matter. It's generally always when I'm talking to someone on discord. Also, when checking past backups, I can generally see that the size of one of the previous backups is randomly different. Like, smaller or larger by 1 megabyte, for example.

Is it possible the backup is doing this? It runs compaction before the actual backup (not a setting I can influence but I believe so). Can this be changed ? Could this be the reason? Maybe the db is so huge the backup takes long enough for some issue to occur during it?

My DB is currently 438MB.

 

Offline dartraiden

Re: Latest libmdbx (commit from 25.8.2018)
« Reply #4 on: 08 10 2018, 13:31:37 »
dbx_sqlite just landed in development version.
 

erthink

  • Guest
Re: Latest libmdbx (commit from 25.8.2018)
« Reply #5 on: 26 08 2019, 19:17:33 »
As the main libmdbx developer I would like to comment:
  • The cause of an above described problems (most likely) was the inherited bug in libmdbx, and at end of september 2018 corresponded fix was imported into Miranda-NG source code.
  • libmdbx was used incorrectly inside Miranda-NG, at least until 19 February 2019. Unfortunately, I can't be sure libmdbx used correctly now, since it is quite a broad subject.
    MDBX provides features that need to be optimally used - this is very important for both performance and database size.
  • Source code libmdbx inside Miranda-NG different from the original (isn't a git submodule or git subtree). Therefore, it is difficult for me to give any guarantees.
    diff with stable/0.2 branch of libmdbx: 33 files changed, 512 insertions(+), 1474 deletions(-)
    $ git diff --stat libs/libmdbx/src/
     libs/libmdbx/src/.travis.yml             |   5 +-
     libs/libmdbx/src/Makefile                |   4 +-
     libs/libmdbx/src/README-RU.md            |  89 ++++++++---------
     libs/libmdbx/src/README.md               |  92 +++++++++---------
     libs/libmdbx/src/libmdbx.files           |   2 -
     libs/libmdbx/src/mdbx.h                  |  59 +-----------
     libs/libmdbx/src/packages/rpm/build.sh   |   0
     libs/libmdbx/src/packages/rpm/package.sh |   0
     libs/libmdbx/src/src/bits.h              |  19 ++--
     libs/libmdbx/src/src/defs.h              |  15 +--
     libs/libmdbx/src/src/mdbx.c              | 789 +++++++++++++++++++++++++++++++++++++++++------------------------------------------------------------------------------------------------------------
     libs/libmdbx/src/src/osal.c              |  79 ++++++++-------
     libs/libmdbx/src/src/tools/mdbx_chk.c    | 182 ++++++++++++++---------------------
     libs/libmdbx/src/src/tools/mdbx_load.1   |   7 --
     libs/libmdbx/src/src/tools/mdbx_load.c   |  68 +++----------
     libs/libmdbx/src/src/version.c           |   4 +-
     libs/libmdbx/src/stdafx.cxx              |  10 --
     libs/libmdbx/src/stdafx.h                |  28 ------
     libs/libmdbx/src/test/CMakeLists.txt     |   2 -
     libs/libmdbx/src/test/append.cc          | 131 -------------------------
     libs/libmdbx/src/test/cases.cc           |   2 -
     libs/libmdbx/src/test/config.h           |   4 +-
     libs/libmdbx/src/test/gc.sh              |   6 +-
     libs/libmdbx/src/test/keygen.cc          |  13 +--
     libs/libmdbx/src/test/keygen.h           |   6 +-
     libs/libmdbx/src/test/loop.bat           |  30 +++---
     libs/libmdbx/src/test/main.cc            |  19 +---
     libs/libmdbx/src/test/test.cc            |  84 +++-------------
     libs/libmdbx/src/test/test.h             |  17 ----
     libs/libmdbx/src/test/test.vcxproj       |   2 -
     libs/libmdbx/src/test/ttl.cc             | 137 --------------------------
     libs/libmdbx/src/test/utils.cc           |  15 ++-
     libs/libmdbx/src/test/utils.h            |  66 +------------
     33 files changed, 512 insertions(+), 1474 deletions(-)
    [close]

    diff with master branch of libmdbx: 38 files changed, 2486 insertions(+), 1393 deletions(-)
    $ git diff --stat libs/libmdbx/src/
     libs/libmdbx/src/.travis.yml                 |   1 +
     libs/libmdbx/src/Makefile                    |  73 +++++++++++---
     libs/libmdbx/src/README-RU.md                | 257 ++++++++++++++++++++++++++++++++++++------------
     libs/libmdbx/src/README.md                   | 539 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------------
     libs/libmdbx/src/appveyor.yml                |   2 +-
     libs/libmdbx/src/libmdbx.files               |  25 ++---
     libs/libmdbx/src/libmdbx.includes            |   2 -
     libs/libmdbx/src/mdbx.h                      |  62 +++++++++---
     libs/libmdbx/src/packages/rpm/CMakeLists.txt |   6 +-
     libs/libmdbx/src/packages/rpm/build.sh       |   0
     libs/libmdbx/src/packages/rpm/package.sh     |   0
     libs/libmdbx/src/src/bits.h                  | 311 +++++++++++++++++++++++++++++++---------------------------
     libs/libmdbx/src/src/defs.h                  |  84 ++++++++--------
     libs/libmdbx/src/src/lck-posix.c             | 293 ++++++++++++++++++++++++++++++++-----------------------
     libs/libmdbx/src/src/lck-windows.c           |  13 ++-
     libs/libmdbx/src/src/mdbx.c                  | 773 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----------------------------------------------
     libs/libmdbx/src/src/osal.c                  | 321 +++++++++++++++++++++++++++++++++++++++---------------------
     libs/libmdbx/src/src/osal.h                  | 176 ++++++++++++++++++++++++---------
     libs/libmdbx/src/src/tools/mdbx_chk.c        | 216 +++++++++++++++++++++++++++++------------
     libs/libmdbx/src/stdafx.cxx                  |  10 --
     libs/libmdbx/src/stdafx.h                    |  28 ------
     libs/libmdbx/src/test/append.cc              |  62 +++++++++---
     libs/libmdbx/src/test/base.h                 |   4 +
     libs/libmdbx/src/test/cases.cc               |   3 +-
     libs/libmdbx/src/test/config.cc              |   4 +-
     libs/libmdbx/src/test/config.h               |  20 +++-
     libs/libmdbx/src/test/gc.sh                  |  66 -------------
     libs/libmdbx/src/test/hill.cc                | 205 +++++++++++++++++++++++++++++----------
     libs/libmdbx/src/test/keygen.cc              |   3 +-
     libs/libmdbx/src/test/loop.bat               |  15 ---
     libs/libmdbx/src/test/main.cc                |  26 +++--
     libs/libmdbx/src/test/osal-unix.cc           |  10 +-
     libs/libmdbx/src/test/osal-windows.cc        |  12 ++-
     libs/libmdbx/src/test/test.cc                |  87 ++++++++++++++---
     libs/libmdbx/src/test/test.h                 |   3 +
     libs/libmdbx/src/test/ttl.cc                 | 100 +++++++++++++------
     libs/libmdbx/src/test/utils.cc               |   5 +-
     libs/libmdbx/src/test/utils.h                |  62 +++---------
     38 files changed, 2486 insertions(+), 1393 deletions(-)
    [close]

    The pull-request to fix this.
However, I want to thank both the developers of Miranda-NG (especially @ghazan) and everyone who took part in the testing. In General, this allowed to implement an exclusive mode of working with the database on network drives and significantly advance in testing on the Windows platform.
« Last Edit: 27 08 2019, 10:10:14 by erthink »
 

Offline ghazan

Re: Latest libmdbx (commit from 25.8.2018)
« Reply #6 on: 27 08 2019, 15:47:32 »
erthink,
this difference exists because for Miranda libmdbx was synced with master, not with stable/0.2. if you insist that master shall not be used, I can easily roll back libmdbx to 0.2

also, the code of mdbx_load & mdbx_save utilities cannot be imported directly, because of problems of printing cyrillic error messages from the file system. thus using libmdbx as a subtree isn't possible

also, what is the sense of having libmdbx if we coudn't use it as a shared module? in Windows we aren't afraid of using them
 

Offline dartraiden

Re: Latest libmdbx (commit from 25.8.2018)
« Reply #7 on: 27 08 2019, 15:55:57 »
btw, our biggest problem with libmdbx is crashing under Wine
 

erthink

  • Guest
Re: Latest libmdbx (commit from 25.8.2018)
« Reply #8 on: 28 08 2019, 10:19:29 »
ghazan,
Quote
his difference exists because for Miranda libmdbx was synced with master, not with stable/0.2. if you insist that master shall not be used, I can easily roll back libmdbx to 0.2
git subtree and git submodule not only allow this, but are designed to do so.
moreover, there are too many differences to correlate them with the libmdbx's changelog - this is the trouble.

Anyway, I'd like you to put the code in perfect condition.
Including, later, resolving all of my comments on libmdbx' API usage (as far as I remember there were ones).
This is important for durability, performance and minimize database file size.

Quote
also, the code of mdbx_load & mdbx_save utilities cannot be imported directly, because of problems of printing cyrillic error messages from the file system. thus using libmdbx as a subtree isn't possible
Please fill corresponding issue at https://github.com/leo-yuriev/libmdbx/issues and I will fix it.

Quote
also, what is the sense of having libmdbx if we coudn't use it as a shared module? in Windows we aren't afraid of using them
The problem is not libmdbx, but how you collect and use it.
If I am not mistaken, you need support for very old versions of Windows, which had and have problems with Thread-Local-Storage destructors when unloading DLL.
Due this you build libmdbx in the configuration, which can create problems in some scenarios.
However, currently libmdbx inside Miranda is only used by MDBX-driver which is always loaded and unloaded explicitly. Therefore, the embedding libmdbx as a static libraries only in this driver looks safe and rational decision.

dartraiden,
Quote
btw, our biggest problem with libmdbx is crashing under Wine
Likely that the problem in Wine, not in libmdbx.
However, if you fill an issue at https://github.com/leo-yuriev/libmdbx/issues (with backtrace(s), etc) I will try to fix.
For instance - does works or crashes "mdbx_test.exe basic" in case you build the 'clean' mdbx from the original source code?

« Last Edit: 28 08 2019, 13:07:57 by erthink »
 

Offline ghazan

Re: Latest libmdbx (commit from 25.8.2018)
« Reply #9 on: 28 08 2019, 14:51:29 »
git subtree and git submodule not only allow this, but are designed to do so.
Submodules work awfully in git... sometimes they aren't updated, sometimes they got broken after branch switching...

However, currently libmdbx inside Miranda is only used by MDBX-driver which is always loaded and unloaded explicitly.
no, Miranda also utilises mdbx_load & mdbx_save as the external compactor, and their code also differs from the standard ones.
« Last Edit: 28 08 2019, 14:53:05 by ghazan »
 

Offline ghazan

Re: Latest libmdbx (commit from 25.8.2018)
« Reply #10 on: 28 08 2019, 15:37:45 »
erthink,
ok, merged Miranda with 0.3.2