Database migration

[riki]

Hello, after using the new database driver for about a month, here's a summary of my experience, peppered by a few requests.

(1). The database is locked at all times, so no backup is possible while Miranda is running. This is unfortunate, because of the following point.

(2). Database file handling seems less robust than it used to be. When the computer crashes, the chances of losing the database are quite high. Recently, I lost my data twice on system crashes - which seldom, if ever, happened to me before.

(3). The history occasionally gets filled with duplicate messages. This seems to happen when the Facebook protocol falters, so it might be unrelated to database driver issues. But I'm just guessing.

(4). RAM use: "Menu->Database->Compact" not only reduces database file size, but also significantly reduces RAM use! This happens even when dat file size doesn't change much or at all. My database had reached ~500 MB, but compacting it brought it back to ~189 MB. Now I compact it more or less daily, and the size obviously doesn't change that much anymore, but RAM use will consistently go down to ~60-80 MB, even if it was higher (as in ~100-200 MB). This is very good for 32-bit systems!

Given the above points, I feel two features are needed.

- A mechanism to flush out the database, write a consistent snapshot of it and unlock it. This mechanism could answer to plugins request, such as from database_autobackup plugin, which at the moment doesn't work because it can't copy the locked .dat file. Point (1) above makes it very important to be able to back up frequently. It is possible to tinker with shadow copies, as pointed out by dartraiden, but it's not common to have a backup utility that handles that, and besides, can I be reassured that the temporary shadow copy is consistent when I back it up? I'd much rather have a 100% consistent snapshot produced by Mirand itself when I hit the "backup" button.

- A tool (probably an offline tool as the old db_check.bat) to clean up the database: deleting double messages above all. My histories are cluttered with duplicates, and it's hard/unpractical/dangerous to clean up manually.

I'd really like to thank the developers (Ghazan above all), the beta testers, and the participants to this discussion.

[Vulpix]

1) not sure about this database locked at all times... I'm using the db autobackups plugin and it works just fine. I tried using a previous version of the database from the backup folder to see if the backups are being made correctly - and they are... so maybe you are experiencing some strange bug.

2) I actually asked ghazan about this; he says it should be even more robust than before. I was doing some overclocking and had a system lockup happen several times with miranda running. I never lost any data ( compared it with exportTXT which I have running at all times, no issues).

3) that is true. I also miss the dupe message removal feature...

4) O_o why is your miranda using so much ram? Mine is using about 30MB normally, and when no chats are open and contact list is hidden in tray, the memory usage is around 3-5MB.

[riki]

1) not sure about this database locked at all times... I'm using the db autobackups plugin and it works just fine. I tried using a previous version of the database from the backup folder to see if the backups are being made correctly - and they are... so maybe you are experiencing some strange bug.

Hm, might be. Although, even simple copy from Windows fails with a message saying file is locked. I used to be able to do that.

2) I actually asked ghazan about this; he says it should be even more robust than before. I was doing some overclocking and had a system lockup happen several times with miranda running. I never lost any data ( compared it with exportTXT which I have running at all times, no issues).

Indeed, simple good sense led me, too, to think the new driver should be more stable and robust, so maybe it was just bad luck? Twice in a fortnight is a strong statistical indicator, though.

4) O_o why is your miranda using so much ram? Mine is using about 30MB normally, and when no chats are open and contact list is hidden in tray, the memory usage is around 3-5MB.

I don't know. Might be an operating system thing (caching and whatnot)? With a freshly booted OS and a freshly launched Miranda, chats closed and contact list hidden (as it is now, for example), usage is around 50MB.

Thanks, as usual, for taking the time to reply point by point 

[Vulpix]

1) I can copy it in windows just fine (just tried). Certainly something wrong with your miranda

4) My miranda has been running for 3 days now and such is its ram usage (the low amounts I mentioned)

[Goraf]

3) FB plugin related.

4) Could be that some plugin is leaking memory.

1) Do simple test:

• Download the latest development Miranda archive from the site
• Unpack it somewhere where system shouldn't complain, launch it and create a test account
• Using "Available components list" from main menu add database_autobackup plugin and nothing more
• Restart it and after that try to do backup of this test profile using both of options from main menu: "Backup profile" and "Save profile as..." and see what will happen

EDIT
Try to do it with your normal Miranda being shutdown

[AnrDaemon]

4) O_o why is your miranda using so much ram? Mine is using about 30MB normally, and when no chats are open and contact list is hidden in tray, the memory usage is around 3-5MB.

I wonder how do you count.
Mine's using anywhere from 300 to 500MB.

[Vulpix]

I dunno, see for yourself...

Miranda running for a day; no chats open and contact list minimized (note the somewhat high network usage - caused by tox protocol :p) :



No chats open and contact list open



And clist+a few chat tabs open... with a conversation going



I dunno, should I be counting it differently?

[AnrDaemon]

Oh, you trust the taskmanager?
Then indeed, mine shows 42MB as well there.
Can I see your working set+private bytes ?

[Vulpix]

Same three scenarios, I dunno, looks the same to me.

Everything closed and minimized clist:



Clist visible but no chats open:



Bunch of chats open:



Not like any of these values actually represent the exact RAM usage, so might as well stick with what task manager says.

This is with connected and loaded tox, irc, steam, jabber and discord. And a bunch of plugins.

VersionInfo

CPU: AMD Ryzen 7 1700 Eight-Core Processor  [AMD64 Family 23 Model 1 Stepping 1] [DEP Enabled] [16 CPUs]
Installed RAM: 32697 MBytes
Microsoft Windows 10 Professional Edition, 64-bit (build 15063)
Internet Explorer: 9.11.15063.0 (build 915063)
Administrator privileges: Yes
OS Languages: (UI | Locale (User/System)) : English/English | English/English
Free disk space on Miranda partition: 3416309 MBytes

Miranda NG Version: 0.95.8 alpha build #19884 (555407743e) [running inside WOW64]
Build time: 25 Apr 2018 2:00:38
Profile: shhh secret (MDBX database driver)
Profile size: 442499072 Bytes
Profile creation date: 21 Apr 2018 9:49:41
Language pack: No language pack installed
Service Mode: No

Active Plugins (37):
¤ AvatarHistory.dll v.0.1.0.2 [11 Apr 2018 14:33:04] - Avatar history
¤ AVS.dll v.0.98.1.1 [25 Apr 2018 2:00:42] - Avatar service
¤ Clist_modern.dll v.0.9.1.7 [11 Apr 2018 14:32:46] - Modern contact list
¤ Console.dll v.0.0.7.4 [17 Jan 2018 19:46:52] - Console
¤ CrashDumper.dll v.0.1.0.1 [19 Apr 2018 21:31:32] - Crash dumper
¤ Db_autobackups.dll v.0.0.0.11 [21 Apr 2018 2:01:42] - Db autobackuper
¤ DbEditorPP.dll v.4.0.1.2 [11 Apr 2018 14:32:50] - Database editor++
¤ Dbx_mdbx.dll v.0.95.8.7 [25 Apr 2018 2:01:56] - Miranda NG MDBX database driver
¤ Dbx_mmap.dll v.0.95.8.1 [25 Apr 2018 2:00:42] - Miranda NG mmap database driver
¤ Discord.dll v.0.5.1.2 [25 Apr 2018 2:01:46] - Discord protocol
¤ Dummy.dll v.0.2.0.0 [25 Apr 2018 2:01:48] - Dummy protocol
¤ Fingerprint.dll v.0.1.0.0 [11 Apr 2018 14:33:00] - Fingerprint NG
¤ Folders.dll v.0.2.0.2 [22 Feb 2018 3:00:34] - Custom profile folders
¤ historypp.dll v.1.5.1.5 [18 Apr 2018 22:19:56] - History++
¤ HistoryStats.dll v.0.2.1.1 [11 Apr 2018 14:33:40] - HistoryStats
¤ ICQ.dll v.0.95.8.1 [25 Apr 2018 2:00:46] - IcqOscarJ protocol
¤ IEView.dll v.1.5.0.1 [19 Apr 2018 21:31:18] - IEView
¤ Import.dll v.0.95.8.6 [25 Apr 2018 2:00:56] - Import contacts and messages
¤ ImportTXT.dll v.0.0.1.12 [11 Apr 2018 14:34:40] - Import TXT
¤ IRC.dll v.0.95.7.2 [25 Apr 2018 2:00:46] - IRC protocol
¤ Jabber.dll v.0.11.0.5 [25 Apr 2018 2:01:00] - Jabber protocol
¤ Msg_Export.dll v.3.1.2.1 [11 Apr 2018 14:32:54] - Message export
¤ PluginUpdater.dll v.0.1.2.5 [13 Apr 2018 12:18:40] - Plugin updater
¤ PManagerEx.dll v.0.0.0.5 [11 Apr 2018 14:32:54] - Miranda NG profile changer
¤ Rate.dll v.0.0.2.1 [11 Apr 2018 14:32:32] - Contact`s rate
¤ RecentContacts.dll v.0.1.0.3 [11 Apr 2018 14:32:40] - Recent contacts
¤ Restart.dll v.0.0.6.0 [17 Jan 2018 19:47:26] - Restart
¤ SkypeWeb.dll v.0.12.2.5 [25 Apr 2018 2:01:56] - Skype protocol (Web)
¤ SmileyAdd.dll v.0.3.0.1 [19 Apr 2018 21:31:30] - SmileyAdd
¤ StartPosition.dll v.0.3.0.4 [19 Apr 2018 21:31:48] - Start position
¤ StatusManager.dll v.1.1.0.4 [11 Apr 2018 14:33:28] - Status manager
¤ Steam.dll v.0.11.6.3 [25 Apr 2018 2:01:42] - Steam protocol
¤ TabSRMM.dll v.3.5.0.9 [19 Apr 2018 21:31:12] - TabSRMM
¤ TopToolBar.dll v.0.8.0.4 [11 Apr 2018 14:32:40] - TopToolBar
¤ Tox.dll v.0.11.3.2 [25 Apr 2018 2:02:06] - Tox protocol
¤ UInfoEx.dll v.0.8.5.0 [25 Apr 2018 2:01:02] - User info ext
¤ Variables.dll v.0.2.3.11 [11 Apr 2018 14:32:50] - Variables

Protocols and Accounts:
-------------------------------------------------------------------------------
Discord                  1 - Enabled 0 - Disabled  Loaded
Dummy                    1 - Enabled 0 - Disabled  Loaded
ICQ                      1 - Enabled 0 - Disabled  Loaded
IRC                      1 - Enabled 0 - Disabled  Loaded
JABBER                   1 - Enabled 0 - Disabled  Loaded
SKYPE                    0 - Enabled 1 - Disabled  Loaded
STEAM                    1 - Enabled 0 - Disabled  Loaded
TOX                      1 - Enabled 0 - Disabled  Loaded
MetaContacts             1 - Enabled 0 - Disabled  Not Loaded

Icon Packs:
-------------------------------------------------------------------------------
 Fp_icons.dll [11 Apr 2018 14:34:38]
 HistoryPP_icons.dll [19 Apr 2018 21:32:32]
 Proto_AIM.dll [12 Dec 2017 5:06:28]
 Proto_Discord.dll [25 Apr 2018 2:02:20]
 Proto_Dummy.dll [25 Apr 2018 2:02:18]
 Proto_Facebook.dll [24 Oct 2017 16:35:34]
 Proto_GG.dll [24 Oct 2017 16:35:36]
 Proto_ICQ.dll [25 Apr 2018 2:02:18]
 Proto_IRC.dll [25 Apr 2018 2:02:18]
 Proto_Jabber.dll [25 Apr 2018 2:02:18]
 Proto_MetaContacts.dll [25 Apr 2018 2:02:18]
 Proto_MSN.dll [24 Oct 2017 16:35:36]
 Proto_Skype.dll [25 Apr 2018 2:02:18]
 Proto_Steam.dll [25 Apr 2018 2:02:18]
 Proto_Tox.dll [25 Apr 2018 2:02:18]
 Proto_Twitter.dll [24 Oct 2017 16:35:36]
 Proto_Yahoo.dll [23 Sep 2016 4:01:40]
 TabSRMM_icons.dll [19 Apr 2018 21:32:34]
 Toolbar_icons.dll [11 Apr 2018 14:34:36]
 Twitch.7z [14 Jan 2018 22:41:34]
 UInfoEx_icons.dll [25 Apr 2018 2:02:18]
 xStatus_ICQ.dll [25 Apr 2018 2:02:18]
 xStatus_Jabber.dll [25 Apr 2018 2:02:18]

[close]

[Goraf]

AnrDaemon, like that?



But yes, task manager will show about 10x less 

[ghazan]

AnrDaemon,
rtfm: working set size means nothing, it's just an amount of memory used to cache your private commit

[AnrDaemon]

Yes, of course. Except when it grows uncontrollable, TaskManager happily reports 50% free physical ram while applications are stuck in the swapfile.

[dartraiden]

Since dev version -> new stable, please post all profile conversion bugs here or to GitHub.