Miranda NG Official Community Forum

Forum for English speaking Miranda NG users => News => Topic started by: dartraiden on 23 03 2018, 20:17:09

Title: Database migration
Post by: dartraiden on 23 03 2018, 20:17:09
Attention to all nightly build users.

In the ongoing nightly build, the current dbx_mmap (https://miranda-ng.org/p/Dbx_mmap) database driver will be finally postponed and replaced with the brand new libmdbx driver (https://miranda-ng.org/p/Dbx_mdbx).

You will get a message on startup This database is in old format that isn't supported anymore. Press Yes to convert it to the new format or No to return back .

You need to have libmdbx driver (https://miranda-ng.org/p/Dbx_mdbx) and  Import (https://miranda-ng.org/p/Import) installed, if not, an error message will pount you to the given links.

The conversion should be transparent, also the old database will be backed up. Though it seems reasonable that you make your own backup of Miranda profile before the upgrade.

Any feedback (https://github.com/miranda-ng/miranda-ng/issues/new) about problems with the new profile is greatly appreciated.
Title: Re: Database migration
Post by: ladyzaba on 24 03 2018, 15:43:33
Just one thing, I'm not sure if there's a connection between the migration and this little bug, but today, after this migration, the icon on my Main panel has changed (I use my favourite status icon pack for all protocols and for the Main panel – all of them are unchanged but this one) to the blue Miranda logo and I'm not able to to change it back :-(.
Title: Re: Database migration
Post by: yvesson on 25 03 2018, 09:50:23
Hej,
Seems a lot of my options have disappeared : I can't right clic my contact list for the menu, and there is that "view mode" which appeared and I can't seem to get rid of.
Title: Re: Database migration
Post by: Goraf on 25 03 2018, 11:34:43
yvesson, 'view mode' can be disabled by "Main menu -> Frames -> View modes -> View modes". Do you want to say that it doesn't work or you couldn't found it?
Title: Re: Database migration
Post by: yvesson on 25 03 2018, 14:42:11
Hej,
I don't have "frames" anymore in my menu. :/
Also the option to right clic the contact list to open the options has disappeared too ?
Title: Re: Database migration
Post by: Goraf on 25 03 2018, 18:32:04
yvesson, what contact list do you use? Post a version info. In "Options -> Customize -> Menus" are options for menu items. You can check there if you have them in main menu and if they are checked or unchecked.
Title: Re: Database migration
Post by: yvesson on 25 03 2018, 18:46:44
Hej,
I use modern CL 0.9.1.7, sorry I don't have time to post more atm and won't use mimi until next friday.
Nothing is unchecked in my options, of course I checked that. Moreover, Miranda crashes very often when I look in this part of the options. I think I'll create a new profile and import just my contacts and history, and not my settings, and just re-tweak everything like I want.

Thanks anyway.

Title: Re: Database migration
Post by: ghazan on 25 03 2018, 21:23:02
Quote from: yvesson on 25 03 2018, 14:42:11
I don't have "frames" anymore in my menu.
yes, confirmed.
Title: Re: Database migration
Post by: ditomov on 26 03 2018, 09:12:53
Hi, after today's update, my metacontacts are bugged - all appear in contact list (clist nicer 0.9.2.4) as offline and named '(Unknown contact)'. Renaming does nothing. Reordering does nothing. Clicking on a metacontact in contact list does nothing. Subcontacts show correct status icons (in the subcontacts submenu) and I do receive and send messages to subcontacts.
Title: Re: Database migration
Post by: ditomov on 26 03 2018, 10:00:53
It seems the problem is with the db conversion. I reverted back to mmap 0.95.3.1 (and backup profile, of course) and it works with the latest miranda64.exe and clist_nicer.
Title: Re: Database migration
Post by: riki on 26 03 2018, 10:47:48
Hello, I also updated and got the database converted automatically. I've got no error during the conversion.

Now Miranda seems to work, but there are some problems/strange things:

- I receive an error on every startup: "You can runly only one instance of CmdLine plugin".

- my profile.dat still has yesterday's modification date (the time of the update). Settings don't stick: I disabled the CmdLine plugin because of the error message, but it still remains active (and I get the error).

- sent messages don't go into history

- Miranda seems unable to receive new messages from Facebook. Also load history doesn't work anymore (it did work recently). I have incoming messages that show in the browser but not in Miranda.


I tried to update again, hoping to solve the problem. Nothing seems to have changed.

However, now updater says there are the following plugins to update, even if I do update they still show in the updater next time:

Plugins\Dbx_mdbx.dll
Plugins\Dbx_mmap.dll

I backed up my profile. Do you think I should revert to previous version temporarily? If so, how can I do that?
Title: Re: Database migration
Post by: ghazan on 26 03 2018, 14:50:35
Quote from: ditomov on 26 03 2018, 09:12:53
Hi, after today's update, my metacontacts are bugged - all appear in contact list (clist nicer 0.9.2.4) as offline
all clist problems must have been fixed, new build is already uploaded
Title: Re: Database migration
Post by: ghazan on 26 03 2018, 14:51:53
Quote from: riki on 26 03 2018, 10:47:48
Now Miranda seems to work, but there are some problems/strange things:
it seems that you have another instance of Miranda running into background. try to kill that process, update Miranda to the recent build and retry the import
Title: Re: Database migration
Post by: riki on 26 03 2018, 16:29:30
Your remote diagnosis is spot on. You're a wizard  :o
Problem solved, thanks Ghazan!!  :DRINK:
Title: Re: Database migration
Post by: amix on 27 03 2018, 13:16:38
Hi,
after the automatic database migration, yesterday, many things were chaotic and messed up here (especially contact-list related). I quit and restarted the app today, accepting all updates but upon restart, I got the attached dialogs (named in sequence in which they appeared). Here is the translation:

1. Order Number 493 not found in DLL
2. Can not open profile. Could not find matching database driver.

I quit, restarted again, but this time I got the message, that my profile is invalid. Also, when I start now, the Profile creation wizard pops up and shows my old profile as "Unknown format".

Thanks.

My plugin list (what other info do you need?):

WhoUsesMyFiles.dll
BASS/
Quotes/
AssocMgr.dll
AuthState.dll
AVS.dll
BasicHistory.dll
BuddyExpectator.dll
Clist_modern.dll
Clist_nicer.dll
CloudFile.dll
CrashDumper.dll
Dbx_mdbx.dll
Dbx_mmap.dll
Facebook.dll
GmailNotifier.dll
historypp.dll
ICQ.dll
Import.dll
IRC.dll
Jabber.dll
MirOTR.dll
MobileState.dll
mRadio.dll
MSN.dll
MyDetails.dll
NewsAggregator.dll
PluginUpdater.dll
Quotes.dll
SendSS.dll
SMS.dll
StatusManager.dll
Steam.dll
StopSpamMod.dll
TabSRMM.dll
TopToolBar.dll
Twitter.dll
Title: Re: Database migration
Post by: riki on 28 03 2018, 16:59:49
Luckily I have no problems with contact list etc., but I'm refraining from updates at the moment, since I am always offered Dbx_mdbx.dll and I'm afraid I will have the "uncompatible database" error (it happened once, then restarting resolved it).

There is still a thing. It may be important or not. I am reporting it for the developers.

My database .dat file modification date is still 26 March. Messaging works OK, but the date does not get updated when I send or receive messages. Even closing and reopening Miranda does not change that date. Strange.

P.S. After Ghazan's suggestion, I always check I only have at most 1 Miranda process active in Task Manager, and I killed all leftover Miranda processes after exiting.
Title: Re: Database migration
Post by: dartraiden on 28 03 2018, 20:09:04
>  I always check I only have at most 1 Miranda process active

Not need anymore, we fixed this. Just update to last version and you cannot launch second Miranda with same profile.

Dbx_mdbx got some fixes (and will surely get many more), so it is offered as an update.
Title: Re: Database migration
Post by: ghazan on 28 03 2018, 20:20:21
amix,
you have old Libs/mir_app.mir, that wasn't updated. I recommend you either to download the archive of Miranda's core (cause you cannot launch the profile) from https://miranda-ng.org/distr/x32/miranda32.zip (or https://miranda-ng.org/distr/x64/miranda64.zip for 64-bit version) and install all libs from it, then retry to import your old profile saved as *.BAK: rename it to *.dat and launch Miranda again
Title: Re: Database migration
Post by: SpinalBlood on 29 03 2018, 00:14:59
I tried it some days ago but I have two problems:

- if I try to create a brand new profile with the last nightly of miranda: I can't create any account, it wont show any plugins

- I tried to convert my profile. But when it tried to import the history (during the conversion), many errors appeared
(I don't have a screen right now, I tested it the other day... I should test this again...)

(Version info is only related to the first problem)


VersionInfo
Quote
CPU: Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz [Intel64 Family 6 Model 58 Stepping 9] [DEP Enabled] [4 CPUs]
Installed RAM: 3792 MBytes
Microsoft Windows 7 Ultimate Edition, 64-bit Service Pack 1 (build 7601)
Internet Explorer: 9.11.9600.17801 (build 99600)
Administrator privileges: Yes
OS Languages: (UI | Locale (User/System)) : Italian/Italian | Italian/Italian
Free disk space on Miranda partition: 388163 MBytes

Miranda NG Version: 0.95.8 alpha build #19601 (e6e175b99a) [running inside WOW64]
Build time: 29 Mar 2018 1:00:48
Profile: D:\New\mir958\Profiles\test\test.dat (MDBX database driver)
Profile size: 262144 Bytes
Profile creation date: 29 Mar 2018 1:15:47
Language pack: No language pack installed
Service Mode: No

Active Plugins (13):
¤ AVS.dll v.0.95.3.4 [29 Mar 2018 1:00:52] - Avatar service
¤ Clist_modern.dll v.0.9.1.7 [29 Mar 2018 1:01:16] - Modern contact list
¤ CrashDumper.dll v.0.1.0.1 [29 Mar 2018 1:01:23] - Crash dumper
¤ DbEditorPP.dll v.4.0.1.2 [29 Mar 2018 2:01:22] - Database editor++
¤ Dbx_mdbx.dll v.0.95.8.6 [29 Mar 2018 1:02:11] - Miranda NG MDBX database driver
¤ Dbx_mmap.dll v.0.95.8.1 [29 Mar 2018 1:01:02] - Miranda NG mmap database driver
¤ Facebook.dll v.0.4.1.1 [29 Mar 2018 1:01:20] - Facebook
¤ historypp.dll v.1.5.1.5 [28 Mar 2018 19:15:08] - History++
¤ Import.dll v.0.95.8.5 [29 Mar 2018 1:01:07] - Import contacts and messages
¤ PluginUpdater.dll v.0.1.2.5 [29 Mar 2018 1:01:34] - Plugin updater
¤ Popup.dll v.2.1.1.12 [29 Mar 2018 2:01:06] - Popup plus
¤ TabSRMM.dll v.3.5.0.8 [29 Mar 2018 1:00:59] - TabSRMM
¤ TopToolBar.dll v.0.8.0.4 [29 Mar 2018 1:01:12] - TopToolBar

Protocols and Accounts:
-------------------------------------------------------------------------------
Facebook                 0 - Enabled 0 - Disabled  Loaded
GG                       0 - Enabled 0 - Disabled  Loaded
ICQ                      0 - Enabled 0 - Disabled  Loaded
IRC                      0 - Enabled 0 - Disabled  Loaded
JABBER                   0 - Enabled 0 - Disabled  Loaded
MSN                      0 - Enabled 0 - Disabled  Loaded
MetaContacts             1 - Enabled 0 - Disabled  Loaded

Icon Packs:
-------------------------------------------------------------------------------
 HistoryPP_icons.dll [29 Mar 2018 2:03:04]
 Proto_Facebook.dll [29 Mar 2018 1:03:00]
 Proto_GG.dll [29 Mar 2018 1:03:01]
 Proto_ICQ.dll [29 Mar 2018 1:03:00]
 Proto_IRC.dll [29 Mar 2018 1:03:00]
 Proto_Jabber.dll [29 Mar 2018 1:03:00]
 Proto_MetaContacts.dll [29 Mar 2018 1:03:02]
 Proto_MSN.dll [29 Mar 2018 1:03:01]
 TabSRMM_icons.dll [29 Mar 2018 1:03:02]
 Toolbar_icons.dll [29 Mar 2018 1:03:02]
 xStatus_ICQ.dll [29 Mar 2018 1:03:01]
 xStatus_Jabber.dll [29 Mar 2018 1:03:02]
[close]
Title: Re: Database migration
Post by: dartraiden on 29 03 2018, 00:19:52
Confirmed, https://github.com/miranda-ng/miranda-ng/issues/1229

Please, test history import again, because we fixed a lot of bugs in last days.
Title: Re: Database migration
Post by: SpinalBlood on 29 03 2018, 00:34:36
Ok I will soon

Should be noted that I can't grant the "new account" problem was present some days ago, because in this case I tested it just now
Title: Re: Database migration
Post by: amix on 29 03 2018, 18:08:37
Quote from: ghazan on 28 03 2018, 20:20:21
[...] then retry to import your old profile saved as *.BAK: rename it to *.dat and launch Miranda again

Thanks for help.

Not quite sure if I follow correctly:

After installation of all libs from Core (done), I go to my profiles folder. This is how it looks like:

Code: [Select]
AvatarCache
Avatars
Logs
MirOTR
tabSRMM
amix@DESKTOP.dat
amix@DESKTOP.dat.bak
amix@DESKTOP.dat-lck
I then did:

1. delete *.dat
2. rename *.bak to *.dat
3. start Miranda-NG and let it convert the profile

I just did that. I got a lot of dialogs with "Order Number xyz not found in (several plugins, like ICQ, IRC, Steam, Facebook, etc.)" After that, Miranda started, but no contacts were available in my contacts-list. Only the folders/groups I had. Now the profile is 512KB, before it was 18.816KB. Also, it seems, that none of the plugins I have got loaded.

When I start out with a new profile, I get the choice between:
dbx mmap and MDBX drivers. Which one shall I take? Also, if I select "Start in service mode with import.dll" I can not continue, just quit.

I then decided to delete all from Miranda NG folder and copy all from the archive you suggested into it. Sort of a fresh install. I also did delete the whole profile folder. But then, upon profile creation, chosing the MDBX driver, I got a dialog: Error 1. And a second dialog: No suitable database driver found.
This is what I have now (Windows pops up warning me, that the application would be unsigned/insecure, which it did not do before):
Code: [Select]
C:\Program Files\Miranda NG\
├── Core
│   ├── StdAutoAway.dll
│   ├── StdAway.dll
│   ├── StdClist.dll
│   ├── StdCrypt.dll
│   ├── StdEmail.dll
│   ├── StdFile.dll
│   ├── StdIdle.dll
│   ├── StdMsg.dll
│   ├── StdSSL.dll
│   ├── StdUIHist.dll
│   ├── StdUserInfo.dll
│   └── StdUserOnline.dll
├── Icons
│   └── Proto_MetaContacts.dll
├── Libs
│   ├── api-ms-win-core-console-l1-1-0.dll
│   ├── api-ms-win-core-datetime-l1-1-0.dll
│   ├── api-ms-win-core-debug-l1-1-0.dll
│   ├── api-ms-win-core-errorhandling-l1-1-0.dll
│   ├── api-ms-win-core-file-l1-1-0.dll
│   ├── api-ms-win-core-file-l1-2-0.dll
│   ├── api-ms-win-core-file-l2-1-0.dll
│   ├── api-ms-win-core-handle-l1-1-0.dll
│   ├── api-ms-win-core-heap-l1-1-0.dll
│   ├── api-ms-win-core-interlocked-l1-1-0.dll
│   ├── api-ms-win-core-libraryloader-l1-1-0.dll
│   ├── api-ms-win-core-localization-l1-2-0.dll
│   ├── api-ms-win-core-memory-l1-1-0.dll
│   ├── api-ms-win-core-namedpipe-l1-1-0.dll
│   ├── api-ms-win-core-processenvironment-l1-1-0.dll
│   ├── api-ms-win-core-processthreads-l1-1-0.dll
│   ├── api-ms-win-core-processthreads-l1-1-1.dll
│   ├── api-ms-win-core-profile-l1-1-0.dll
│   ├── api-ms-win-core-rtlsupport-l1-1-0.dll
│   ├── api-ms-win-core-string-l1-1-0.dll
│   ├── api-ms-win-core-synch-l1-1-0.dll
│   ├── api-ms-win-core-synch-l1-2-0.dll
│   ├── api-ms-win-core-sysinfo-l1-1-0.dll
│   ├── api-ms-win-core-timezone-l1-1-0.dll
│   ├── api-ms-win-core-util-l1-1-0.dll
│   ├── api-ms-win-crt-conio-l1-1-0.dll
│   ├── api-ms-win-crt-convert-l1-1-0.dll
│   ├── api-ms-win-crt-environment-l1-1-0.dll
│   ├── api-ms-win-crt-filesystem-l1-1-0.dll
│   ├── api-ms-win-crt-heap-l1-1-0.dll
│   ├── api-ms-win-crt-locale-l1-1-0.dll
│   ├── api-ms-win-crt-math-l1-1-0.dll
│   ├── api-ms-win-crt-multibyte-l1-1-0.dll
│   ├── api-ms-win-crt-private-l1-1-0.dll
│   ├── api-ms-win-crt-process-l1-1-0.dll
│   ├── api-ms-win-crt-runtime-l1-1-0.dll
│   ├── api-ms-win-crt-stdio-l1-1-0.dll
│   ├── api-ms-win-crt-string-l1-1-0.dll
│   ├── api-ms-win-crt-time-l1-1-0.dll
│   ├── api-ms-win-crt-utility-l1-1-0.dll
│   ├── FreeImage.mir
│   ├── libeay32.mir
│   ├── libjson.mir
│   ├── mir_app.mir
│   ├── mir_core.mir
│   ├── msvcp140.dll
│   ├── Pcre16.mir
│   ├── sqlite3.mir
│   ├── ssleay32.mir
│   ├── ucrtbase.dll
│   ├── vcruntime140.dll
│   └── Zlib.mir
├── Miranda64.exe
└── Plugins
    ├── Dbx_mdbx.dll
    ├── Dbx_mmap.dll
    └── Import.dll
Title: Re: Database migration
Post by: ghazan on 30 03 2018, 13:09:32
amix,
Quote from: amix on 29 03 2018, 18:08:37
I got a lot of dialogs with "Order Number xyz not found in (several plugins, like ICQ, IRC, Steam, Facebook, etc.)"
it means that you have different versions of plugins. what do you really need is to run Plugin Updater once on that new profile without accounts, replace all dlls with these proper versions and only then retry the import. evidently, if your protocol plugins aren't loaded, Miranda can't perform the import correctly

PS: if Plugin Updater itself fails to load, you need to download it manually from https://miranda-ng.org/p/PluginUpdater -> development version
Title: Re: Database migration
Post by: yvesson on 30 03 2018, 16:49:37
Quote from: ghazan on 26 03 2018, 14:50:35
all clist problems must have been fixed, new build is already uploaded

Hej,
Seems everything is working fine now, used the backup of my old profile and converted it, now it's like before.
So thanks !
Title: Re: Database migration
Post by: riki on 31 03 2018, 22:36:48
After forcefully rebooting the computer, now Miranda doesn't start correctly. It gives me this error:

Code: [Select]
Miranda was unable to open 'profile.dat', it's in an unknown format.
I tried to relaunch Miranda with no profiles, but the import failed. So I tried updating (update worked OK), but even after updating I wasn't able to import the profile. Import says something like "database read problem".

DbChecker.bat gives me the same error,  "Miranda was unable to open..."

I am worrying. It's my historical profile, with data from over 10 years ago (taken over from the old Miranda-IM). Any suggestions?

(EDIT) A couple of questions for the developers.

- Is the new database format/driver more fragile? I don't remember having such a data disaster (apparent total loss of DB info) with power outages or forced reboots.

- Is a database repair tool scheduled? I used it quite often. Especially with facebook plugin, there were often issues with duplicated messages etc., and that tool solved all such problems for me.
Title: Re: Database migration
Post by: ghazan on 02 04 2018, 12:14:56
riki,

1. DbChecker.bat is totally obsoleted, as well the whole dbchecker plugin

2. Your previous profile was renamed to .bak and stored in the same profile.

3. You can always rename it back and retry the import
Title: Re: Database migration
Post by: riki on 02 04 2018, 20:20:08
Thanks for replying, Ghazan.

Quote from: ghazan on 02 04 2018, 12:14:56
3. You can always rename it back and retry the import

I'd already tried that, but it seems import plugin doesn't like it. Error reading source file. I can open older backups that I made myself, with no issue (no need to import either - Miranda just starts OK).

Sorry, more questions. I hope you don't mind.

How can I know which db driver Miranda is using? Should I switch? How?

By the way, the dat-lck file doesn't get deleted, even after Miranda exits. Is this normal?
Title: Re: Database migration
Post by: dartraiden on 02 04 2018, 21:36:13
Quote
How can I know which db driver Miranda is using?
If CrashDumper plugin is installed: Main menu -> Version information -> Show. You will see smth like this:
Quote
Miranda NG Version: 0.95.8 alpha build #19651 (8d3083580d) x64
Build time: 2 Apr 2018 19:56:00
Profile: C:\Users\dr\AppData\Roaming\Miranda NG\home\home.dat (MDBX database driver <- current profile's database driver)

If you still on MMAP (old format):
- exit Miranda
- make sure no Miranda copies are launched
- backup all Miranda folder
- backup profile (blabla.dat)
- start Miranda, update to the last version via plugin updater and Miranda ask about conversion.

If you already on MDBX and can see this in Version Info, so you already successfully converted profile and use it.

Quote
Is this normal?
Yes, if your profile is MDBX. If MMAP - delete lck manually.
Title: Re: Database migration
Post by: riki on 03 04 2018, 12:46:14
Hello dartraiden, thank you! I checked as you suggested and I'm on MDBX. I think I have been for a long time, since my updater is set up for experimental versions.

I haven't been able to recover my latest profile.dat, however. Whatever happened, I am taking my safety measures, so I set up a daily rsync to back it up.

However, I found that profile.dat file is locked while Miranda is running, so rsync can't read it to back it up, and the backup fails. The only way I found to make it work is to exit Miranda before the automatic backup time. This is unconvenient, as Miranda is always running on my system.

I wonder: could a command be added to flush and unlock profile.dat? If so, I could schedule it inside of Miranda and the daily backup could proceed without errors. Or maybe some other way to make profile.dat accessible by external processes when needed?
Title: Re: Database migration
Post by: AnrDaemon on 08 04 2018, 12:59:23
Many archiving programs, such as RAR, could open write-locked files.
But keep in mind that such method is prone to making an inconsistent backup.
Your best option is something like https://wiki.miranda-ng.org/index.php?title=Plugin:Db_autobackups
Haven't used it myself, no idea if it's working with current database driver.
Title: Re: Database migration
Post by: riki on 08 04 2018, 13:42:47
Quote from: AnrDaemon on 08 04 2018, 12:59:23
Many archiving programs, such as RAR, could open write-locked files.
But keep in mind that such method is prone to making an inconsistent backup.

That's why I asked if it's possible to flush the database file and leave it in a consistent state, just for the sake of backup, without stopping Miranda.

Quote
Your best option is something like https://wiki.miranda-ng.org/index.php?title=Plugin:Db_autobackups
Haven't used it myself, no idea if it's working with current database driver.

Good advice, AnrDaemon. I'll install this plugin and see. The download page says it should work with the mdbx database driver. However, there's a warning that enabling the "zip it" option might produce corrupted backups. :(

EDIT 1 - I had to choose the experimental version (the stable version didn't appear in plugin list). It appears to work fine and creates neatly named backups. Haven't verified if the backup files are consistent, though.

EDIT 2 - No, the plugin can't actually copy the database. The backup file gets opened/created, then the copy aborts and the backup gets closed/deleted. I guess it's the same locked profile.dat problem. Nice try anyway  :)
Title: Re: Database migration
Post by: dartraiden on 08 04 2018, 18:39:26
Db_autobackups works perfectly for me.

Can you provide Version Info report and maybe some important info? (for example, "profile located on network drive", "profile path contains non-ascii symbols", etc)

Zip-packing is broken in some circumstances (it produces corrupted archives on my virtual machine with Cyrillic paths and long paths).
Title: Re: Database migration
Post by: riki on 08 04 2018, 19:06:21
Thanks for your interest, dartraiden.

Here's my version info.

https://pastebin.com/gQRQcqUV

There are no special mentions to be made: no non-ISO-Latin characters in pathname, save path for backups is
C:\Documents and Settings\TheUser\Application Data\Miranda\Profiles\default\AutoBackups\
as suggested by the plugin.

What happens is the file gets created (it appears in the window for a short time), then it disappears. Actually, it is the same behavior I observe when I try to manually make a copy in the same directory as profile.dat, and I guess is the same problem that rsync finds when it tries to copy the profile to my remote rsync server.

EDIT - In my version info, the autobackup plugin is disabled. I turned it off myself because it ddn't work after extensive testing. I'll turn it back on now, just to able to report in case of questions from you or some other helping soul.

Title: Re: Database migration
Post by: dartraiden on 08 04 2018, 19:37:28
Profile: C:\Documents and Settings\TheUser\Application Data\Miranda\Profiles\default\profile.dat (MDBX database driver)

 :o

Miranda even shouldn't start if bolded parts differ. This parts MUST be identical.
Title: Re: Database migration
Post by: Goraf on 08 04 2018, 19:41:21
From Edit 1 and previous post seems that you all the time trying to mix up things. Like it was mentioned before to you, it is very important to update plugins only by PluginUpdater. By not doing that you are asking yourself for unpredictable problems. The same applies for installing them. For that purpose you have "Available component list" entry in main menu. Please remember about that in the future
Title: Re: Database migration
Post by: dartraiden on 08 04 2018, 19:49:02
Quote
I had to choose the experimental version (the stable version didn't appear in plugin list)
Just tested on stable
Title: Re: Database migration
Post by: riki on 08 04 2018, 20:27:33
Oops!
Sorry guys, thanks for taking the time for setting me straight. What actually happened is I'd edited the version info to keep my full name from appearing (profile at the moment is named firstname-lastname.dat).

I'd installed the autobackup plugin manually (no available component list, ehm), but I think the installation was correct. The package contained only the dll file. But I only update via updater, always.

Now I removed the autobackup plugin and used "Available Component List" to install it. No choice stable/experimental, maybe that just happened because I went straight to the plugin download page.

Here's the new version info - only last 2 components of profile.dat are slightly edited. Now I also removed the unloadable plugins, the folder is clear of rubbish.

https://pastebin.com/6MY0JSF4

P.S. I will remember to use only "Available Component List" in the future. Thank you.
Title: Re: Database migration
Post by: dartraiden on 08 04 2018, 20:28:26
Quote
No choice stable/experimental
Updater always shows plugins list according to Miranda version (stable plugins for stable version, etc).
Title: Re: Database migration
Post by: dartraiden on 09 04 2018, 12:07:42
Please test your backup utility (rsync?) after next development version update.
https://github.com/miranda-ng/miranda-ng/commit/89ac15679e32fa707757a95fafbfc7ef9b3e705d

Another way: use backup tool which supports shadow copies (https://en.wikipedia.org/wiki/Shadow_Copy)
Title: Re: Database migration
Post by: riki on 10 04 2018, 19:13:33
Thanks for bearing with me, dartraiden.

I've updated Miranda twice since my last message. My versioninfo is here: https://pastebin.com/6mUkUzh6

I've visited the github page you linked, and it seems like it should solve my problem, but unfortunately neither rsync nor Miranda's autobackup plugin are yet able to copy the file while Miranda is running.

Your suggestion about shadow copies is a good one! Setting up the shadow copy is a bit of a pain, as it requires an extra script etc., but I already have the necessary machinery in place to make backups of the registry about once-twice a month. So maybe I should create a new daily backup routine just for my profile.dat, with all the shadow copy mechanism in place. I'll have it launch at a different time from the "normal" daily backup (no shadow copy necessary there), as to avoid conflicts or excess disk stress.

EDIT
After last update (2018-04-13, db drivers and more stuff), the db file still appears to be locked - so no live running backup is possible either by autobackup plugin or rsync without tinkering with shadow copies.

However, RAM use by Miranda has dropped by 30-35%. Wow!  :THUMBS UP:
Title: Re: Database migration
Post by: Vulpix on 13 04 2018, 14:07:50
Most of various db migration issues have been resolved in the past few days thanks to Ghazan; people who have been holding off on migration, please make a backup and try it, see how it works for you. All good over here! :)
Title: Re: Database migration
Post by: sir_qwerty on 16 04 2018, 07:36:28
Quote from: dartraiden on 08 04 2018, 20:28:26
Updater always shows plugins list according to Miranda version (stable plugins for stable version, etc).
The same assumption exactly sent me into the endless loop when updating database and migrating to new db driver in alpha build. I experienced this scenario:
So it seems to me that PluginUpdater doesn't stick with Files source according to core version but according to the setting.

(https://ctrlv.cz/shots/2018/04/16/sErB.png)
Title: Re: Database migration
Post by: Goraf on 16 04 2018, 09:13:46
Quote from: dartraiden on 08 04 2018, 20:28:26
Updater always shows plugins list according to Miranda version (stable plugins for stable version, etc).
Yes, but version you choose. So it should be understood as "according to Miranda version you have, and that version is the one you choose in options"

That's why we have things like (see screenshot) or similar in many places.
That's why we have https://wiki.miranda-ng.org/index.php?title=Plugin:PluginUpdater#Version_choice or https://wiki.miranda-ng.org/index.php?title=Installation_and_update#Updating_Miranda_NG

What do you think we should do more or where when your answer suggest that it is still not 100% visible or clear? We are open on suggestions or advice in this manner.
Title: Re: Database migration
Post by: riki on 17 04 2018, 23:59:20
Hello, trouble here. Miranda starts with an error message. 

Miranda32.exe has encountered a problem and needs to close...


For more information about this error:

AppName: miranda32.exe
AppVer: 0.95.8.19810
ModName: dbx_mdbx.dll


To view technical information about this error report, click here:
(Can't copy/paste the ensuing dialog box. Suggestions?)

Additional information in \Documents and Settings\...\224f_appcompat.txt
I pasted here: https://pastebin.com/SR591yym

If I try to use the commandline MimCmd.exe, I get 
 
Could not create connection with Miranda or could not retrieve list of known com
mands.


I think I lost my database again.  :( Last backup is 3 days old, since I backed up with Miranda stopped, or the file wouldn't copy.

EDIT - It seems that database file size has increased significantly since the update, although my Miranda activity is average as usual.

Last backup before migration (but I think I was already on new driver) - 183,697,408 bytes
First backup after migration (this database won't open in Miranda, it got damaged I think) - 268,435,456 bytes
Last backup after migration (3 days old, opens OK) - 536,870,912 bytes

I also find memory (RAM) use by Miranda to oscillate wildly between ~70 MB and ~600 MB over the hours/half hours.

Is this expected?

(I hope I'm not a nuisance and my reports help improve Miranda)
Title: Re: Database migration
Post by: Vulpix on 18 04 2018, 07:22:29
For database size increasing post migration, this is expected; since we now store indices as well (no longer a linked list like mmap was).

For example my database used to be about 297 MB (properly maintained with no wasted space); post migration it is now 420MB. But it shouldn't randomly, wildly increase afterwards.. it should grow at about a normal pace.
Title: Re: Database migration
Post by: riki on 26 04 2018, 10:53:30
Hello, after using the new database driver for about a month, here's a summary of my experience, peppered by a few requests.

(1). The database is locked at all times, so no backup is possible while Miranda is running. This is unfortunate, because of the following point.

(2). Database file handling seems less robust than it used to be. When the computer crashes, the chances of losing the database are quite high. Recently, I lost my data twice on system crashes - which seldom, if ever, happened to me before.

(3). The history occasionally gets filled with duplicate messages. This seems to happen when the Facebook protocol falters, so it might be unrelated to database driver issues. But I'm just guessing.

(4). RAM use: "Menu->Database->Compact" not only reduces database file size, but also significantly reduces RAM use! This happens even when dat file size doesn't change much or at all. My database had reached ~500 MB, but compacting it brought it back to ~189 MB. Now I compact it more or less daily, and the size obviously doesn't change that much anymore, but RAM use will consistently go down to ~60-80 MB, even if it was higher (as in ~100-200 MB). This is very good for 32-bit systems!

Given the above points, I feel two features are needed.

- A mechanism to flush out the database, write a consistent snapshot of it and unlock it. This mechanism could answer to plugins request, such as from database_autobackup plugin, which at the moment doesn't work because it can't copy the locked .dat file. Point (1) above makes it very important to be able to back up frequently. It is possible to tinker with shadow copies, as pointed out by dartraiden, but it's not common to have a backup utility that handles that, and besides, can I be reassured that the temporary shadow copy is consistent when I back it up? I'd much rather have a 100% consistent snapshot produced by Mirand itself when I hit the "backup" button.

- A tool (probably an offline tool as the old db_check.bat) to clean up the database: deleting double messages above all. My histories are cluttered with duplicates, and it's hard/unpractical/dangerous to clean up manually.

I'd really like to thank the developers (Ghazan above all), the beta testers, and the participants to this discussion.
Title: Re: Database migration
Post by: Vulpix on 26 04 2018, 11:24:25
1) not sure about this database locked at all times... I'm using the db autobackups plugin and it works just fine. I tried using a previous version of the database from the backup folder to see if the backups are being made correctly - and they are... so maybe you are experiencing some strange bug.

2) I actually asked ghazan about this; he says it should be even more robust than before. I was doing some overclocking and had a system lockup happen several times with miranda running. I never lost any data ( compared it with exportTXT which I have running at all times, no issues).

3) that is true. I also miss the dupe message removal feature...

4) O_o why is your miranda using so much ram? Mine is using about 30MB normally, and when no chats are open and contact list is hidden in tray, the memory usage is around 3-5MB.
Title: Re: Database migration
Post by: riki on 26 04 2018, 12:36:16
Quote from: Vulpix on 26 04 2018, 11:24:25
1) not sure about this database locked at all times... I'm using the db autobackups plugin and it works just fine. I tried using a previous version of the database from the backup folder to see if the backups are being made correctly - and they are... so maybe you are experiencing some strange bug.
Hm, might be. Although, even simple copy from Windows fails with a message saying file is locked. I used to be able to do that.

Quote
2) I actually asked ghazan about this; he says it should be even more robust than before. I was doing some overclocking and had a system lockup happen several times with miranda running. I never lost any data ( compared it with exportTXT which I have running at all times, no issues).
Indeed, simple good sense led me, too, to think the new driver should be more stable and robust, so maybe it was just bad luck? Twice in a fortnight is a strong statistical indicator, though.

Quote
4) O_o why is your miranda using so much ram? Mine is using about 30MB normally, and when no chats are open and contact list is hidden in tray, the memory usage is around 3-5MB.
I don't know. Might be an operating system thing (caching and whatnot)? With a freshly booted OS and a freshly launched Miranda, chats closed and contact list hidden (as it is now, for example), usage is around 50MB.

Thanks, as usual, for taking the time to reply point by point  :)
Title: Re: Database migration
Post by: Vulpix on 26 04 2018, 12:48:28
1) I can copy it in windows just fine (just tried). Certainly something wrong with your miranda :o

4) My miranda has been running for 3 days now and such is its ram usage :o (the low amounts I mentioned)
Title: Re: Database migration
Post by: Goraf on 26 04 2018, 15:44:31
3) FB plugin related.

4) Could be that some plugin is leaking memory.

1) Do simple test:

EDIT
Try to do it with your normal Miranda being shutdown
Title: Re: Database migration
Post by: AnrDaemon on 27 04 2018, 01:21:52
Quote from: Vulpix on 26 04 2018, 11:24:25
4) O_o why is your miranda using so much ram? Mine is using about 30MB normally, and when no chats are open and contact list is hidden in tray, the memory usage is around 3-5MB.
I wonder how do you count.
Mine's using anywhere from 300 to 500MB.
Title: Re: Database migration
Post by: Vulpix on 27 04 2018, 06:51:38
I dunno, see for yourself...

Miranda running for a day; no chats open and contact list minimized (note the somewhat high network usage - caused by tox protocol :p) :

(https://i.imgur.com/aZKRHXH.png)

No chats open and contact list open

(https://i.imgur.com/urZZzwS.png)

And clist+a few chat tabs open... with a conversation going

(https://i.imgur.com/54sauSc.png)

I dunno, should I be counting it differently?
Title: Re: Database migration
Post by: AnrDaemon on 27 04 2018, 08:35:46
Oh, you trust the taskmanager? :D
Then indeed, mine shows 42MB as well there.
Can I see your working set+private bytes ?
Title: Re: Database migration
Post by: Vulpix on 27 04 2018, 08:58:52
Same three scenarios, I dunno, looks the same to me.

Everything closed and minimized clist:

(https://i.imgur.com/bUZurGP.png)

Clist visible but no chats open:

(https://i.imgur.com/iqZVJOr.png)

Bunch of chats open:

(https://i.imgur.com/CjnhmrO.png)

Not like any of these values actually represent the exact RAM usage, so might as well stick with what task manager says.

This is with connected and loaded tox, irc, steam, jabber and discord. And a bunch of plugins.

VersionInfo
Quote
CPU: AMD Ryzen 7 1700 Eight-Core Processor  [AMD64 Family 23 Model 1 Stepping 1] [DEP Enabled] [16 CPUs]
Installed RAM: 32697 MBytes
Microsoft Windows 10 Professional Edition, 64-bit (build 15063)
Internet Explorer: 9.11.15063.0 (build 915063)
Administrator privileges: Yes
OS Languages: (UI | Locale (User/System)) : English/English | English/English
Free disk space on Miranda partition: 3416309 MBytes

Miranda NG Version: 0.95.8 alpha build #19884 (555407743e) [running inside WOW64]
Build time: 25 Apr 2018 2:00:38
Profile: shhh secret (MDBX database driver)
Profile size: 442499072 Bytes
Profile creation date: 21 Apr 2018 9:49:41
Language pack: No language pack installed
Service Mode: No

Active Plugins (37):
¤ AvatarHistory.dll v.0.1.0.2 [11 Apr 2018 14:33:04] - Avatar history
¤ AVS.dll v.0.98.1.1 [25 Apr 2018 2:00:42] - Avatar service
¤ Clist_modern.dll v.0.9.1.7 [11 Apr 2018 14:32:46] - Modern contact list
¤ Console.dll v.0.0.7.4 [17 Jan 2018 19:46:52] - Console
¤ CrashDumper.dll v.0.1.0.1 [19 Apr 2018 21:31:32] - Crash dumper
¤ Db_autobackups.dll v.0.0.0.11 [21 Apr 2018 2:01:42] - Db autobackuper
¤ DbEditorPP.dll v.4.0.1.2 [11 Apr 2018 14:32:50] - Database editor++
¤ Dbx_mdbx.dll v.0.95.8.7 [25 Apr 2018 2:01:56] - Miranda NG MDBX database driver
¤ Dbx_mmap.dll v.0.95.8.1 [25 Apr 2018 2:00:42] - Miranda NG mmap database driver
¤ Discord.dll v.0.5.1.2 [25 Apr 2018 2:01:46] - Discord protocol
¤ Dummy.dll v.0.2.0.0 [25 Apr 2018 2:01:48] - Dummy protocol
¤ Fingerprint.dll v.0.1.0.0 [11 Apr 2018 14:33:00] - Fingerprint NG
¤ Folders.dll v.0.2.0.2 [22 Feb 2018 3:00:34] - Custom profile folders
¤ historypp.dll v.1.5.1.5 [18 Apr 2018 22:19:56] - History++
¤ HistoryStats.dll v.0.2.1.1 [11 Apr 2018 14:33:40] - HistoryStats
¤ ICQ.dll v.0.95.8.1 [25 Apr 2018 2:00:46] - IcqOscarJ protocol
¤ IEView.dll v.1.5.0.1 [19 Apr 2018 21:31:18] - IEView
¤ Import.dll v.0.95.8.6 [25 Apr 2018 2:00:56] - Import contacts and messages
¤ ImportTXT.dll v.0.0.1.12 [11 Apr 2018 14:34:40] - Import TXT
¤ IRC.dll v.0.95.7.2 [25 Apr 2018 2:00:46] - IRC protocol
¤ Jabber.dll v.0.11.0.5 [25 Apr 2018 2:01:00] - Jabber protocol
¤ Msg_Export.dll v.3.1.2.1 [11 Apr 2018 14:32:54] - Message export
¤ PluginUpdater.dll v.0.1.2.5 [13 Apr 2018 12:18:40] - Plugin updater
¤ PManagerEx.dll v.0.0.0.5 [11 Apr 2018 14:32:54] - Miranda NG profile changer
¤ Rate.dll v.0.0.2.1 [11 Apr 2018 14:32:32] - Contact`s rate
¤ RecentContacts.dll v.0.1.0.3 [11 Apr 2018 14:32:40] - Recent contacts
¤ Restart.dll v.0.0.6.0 [17 Jan 2018 19:47:26] - Restart
¤ SkypeWeb.dll v.0.12.2.5 [25 Apr 2018 2:01:56] - Skype protocol (Web)
¤ SmileyAdd.dll v.0.3.0.1 [19 Apr 2018 21:31:30] - SmileyAdd
¤ StartPosition.dll v.0.3.0.4 [19 Apr 2018 21:31:48] - Start position
¤ StatusManager.dll v.1.1.0.4 [11 Apr 2018 14:33:28] - Status manager
¤ Steam.dll v.0.11.6.3 [25 Apr 2018 2:01:42] - Steam protocol
¤ TabSRMM.dll v.3.5.0.9 [19 Apr 2018 21:31:12] - TabSRMM
¤ TopToolBar.dll v.0.8.0.4 [11 Apr 2018 14:32:40] - TopToolBar
¤ Tox.dll v.0.11.3.2 [25 Apr 2018 2:02:06] - Tox protocol
¤ UInfoEx.dll v.0.8.5.0 [25 Apr 2018 2:01:02] - User info ext
¤ Variables.dll v.0.2.3.11 [11 Apr 2018 14:32:50] - Variables

Protocols and Accounts:
-------------------------------------------------------------------------------
Discord                  1 - Enabled 0 - Disabled  Loaded
Dummy                    1 - Enabled 0 - Disabled  Loaded
ICQ                      1 - Enabled 0 - Disabled  Loaded
IRC                      1 - Enabled 0 - Disabled  Loaded
JABBER                   1 - Enabled 0 - Disabled  Loaded
SKYPE                    0 - Enabled 1 - Disabled  Loaded
STEAM                    1 - Enabled 0 - Disabled  Loaded
TOX                      1 - Enabled 0 - Disabled  Loaded
MetaContacts             1 - Enabled 0 - Disabled  Not Loaded

Icon Packs:
-------------------------------------------------------------------------------
 Fp_icons.dll [11 Apr 2018 14:34:38]
 HistoryPP_icons.dll [19 Apr 2018 21:32:32]
 Proto_AIM.dll [12 Dec 2017 5:06:28]
 Proto_Discord.dll [25 Apr 2018 2:02:20]
 Proto_Dummy.dll [25 Apr 2018 2:02:18]
 Proto_Facebook.dll [24 Oct 2017 16:35:34]
 Proto_GG.dll [24 Oct 2017 16:35:36]
 Proto_ICQ.dll [25 Apr 2018 2:02:18]
 Proto_IRC.dll [25 Apr 2018 2:02:18]
 Proto_Jabber.dll [25 Apr 2018 2:02:18]
 Proto_MetaContacts.dll [25 Apr 2018 2:02:18]
 Proto_MSN.dll [24 Oct 2017 16:35:36]
 Proto_Skype.dll [25 Apr 2018 2:02:18]
 Proto_Steam.dll [25 Apr 2018 2:02:18]
 Proto_Tox.dll [25 Apr 2018 2:02:18]
 Proto_Twitter.dll [24 Oct 2017 16:35:36]
 Proto_Yahoo.dll [23 Sep 2016 4:01:40]
 TabSRMM_icons.dll [19 Apr 2018 21:32:34]
 Toolbar_icons.dll [11 Apr 2018 14:34:36]
 Twitch.7z [14 Jan 2018 22:41:34]
 UInfoEx_icons.dll [25 Apr 2018 2:02:18]
 xStatus_ICQ.dll [25 Apr 2018 2:02:18]
 xStatus_Jabber.dll [25 Apr 2018 2:02:18]
[close]
Title: Re: Database migration
Post by: Goraf on 27 04 2018, 10:48:25
AnrDaemon, like that?

(https://imgur.com/qRGuO5K.png)

But yes, task manager will show about 10x less  ;D
Title: Re: Database migration
Post by: ghazan on 02 05 2018, 19:31:33
AnrDaemon,
rtfm: working set size means nothing, it's just an amount of memory used to cache your private commit
Title: Re: Database migration
Post by: AnrDaemon on 03 05 2018, 07:56:18
Yes, of course. Except when it grows uncontrollable, TaskManager happily reports 50% free physical ram while applications are stuck in the swapfile.
Title: Re: Database migration
Post by: dartraiden on 04 05 2018, 21:20:38
Since dev version -> new stable, please post all profile conversion bugs here (https://forum.miranda-ng.org/index.php?topic=5461.0) or to GitHub (https://github.com/miranda-ng/miranda-ng/issues/new).