DBCheck corrupts history for Steam and Tox protocol

[Vulpix]

Running database check on the database will corrupt the history:

Before - no corruptions observed.

History check runs and finds "duplicate" items and "skips" them (yellow text) in the dbchecker.

This results in massive corruption of contact's history in the database. Some messages are fine, but mostly they are destroyed and replaced by garbled text.

OWN messages are NOT affected - only history events related to the person you are talking with.

Me, 20.8.2014 9:27:41:
Like button was moved?

<Party B>, 9:27:48:
ꖲ鏪릂ꫨꆃ苮몖越苫ꞗ諮膂韩越苫ꞗ諮膂韩越苫ꞗ諮膂韩趂蟫꒞苪ꚗ髮놲꟫릖苪ꊇ鋮뎢鏪릂ꛩ誨苩袻胧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧ꂴ臧肐胢늕뇦겥胢鎽ꃯ貥ꓡ誨胡ꖺ/⼀ਙ
ꔐȎ

Vulpix, 9:27:56:
Not for me

<Party B>, 9:27:59:
Oh

<Party B>, 9:28:06:
ꞇ鋮뒂蟨떒苪ꞇ鋮뒂蟨떒苪ꞇ鋮뒂蟨떚苨ꊃ髮閊蟫ꆞ닪ꊃ髮붎菪ꂎ단ꆓ苮ꢪ菨肂雪ʺ臢⾯볢馨邔軈ꀀ단ꆓ苮ꢪ菨肂雪ʺ諮膂韩越苫ꞗ諮膂韩越苫ꞗ諮膂韩越苫ꞗ苧ꊇ黮袂韨ꖚ닪ꊧ雮貂蟨꾒ꋫꎓ苮ꢦ꣮钂믨肀들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁들떁郦검闦ꆱꗧ꺀뷦ꂠꗣ릤꣠肀뫠ʥ漀⼠

Vulpix, 9:28:15:
Probably not updated? :3

Sometimes this manifests itself by repeating segments of my own message with the party b's timestamp:

Vulpix, 19.8.2014 19:56:11:
I'm going to that quest on my druid, and then we can play~ :3

<party b>, 19:56:41:
I'm going to that quest on my druid, and then we can play~ :3

Vulpix, 20:47:22:
I need to talk to a friend about something so just a moment

<party b>, 20:47:51:
I need to talk to a friend about something so just a moment

Vulpix, 20:55:49:
Alrighty!

Vulpix, 20:55:51:
Let's play~

Vulpix, 22:38:23:
Actually, back when troll racial used to heal the troll (allows spirit-based hp regen to continue in combat or something)

Vulpix, 22:38:39:
People tried to get as much warrior spirit gear to see how much the regen was and it was still absolutely useless x3

<party b>, 22:39:20:
People tried to get as much warrior spirit gear to see how much the regen was and it was still absolutely useless x3

<party b>, 22:39:33:
People tried to get as much warrior spirit gear to see how much the regen was and it was still absolutely useless x3

<party b>, 22:39:42:
People tried to get as much warrior spirit gear to see how much the regen was and it was still absolutely useless x3

Obviously, <party B> didn't repeat my sentences 3 times randomly. They said something normal - but it's gone, and my text is in its place.

The same is observed for Steam plugin.

This issue is not observed for (for example) AIM plugin.

Settings for dbcheck: Aggressive/Mark as read/Convert old (all 3 enabled)

VI:

VersionInfo

CPU: Intel(R) Core(TM) i7-4765T CPU @ 2.00GHz [Intel64 Family 6 Model 60 Stepping 3] [DEP Enabled] [8 CPUs]
Installed RAM: 16329 MBytes
Operating System: Microsoft Windows 7 Ultimate Edition, 64-bit Service Pack 1 (build 7601)
Internet Explorer: 8.0.7601.17514 (build 87601)
Administrator privileges: Yes
OS Languages: (UI | Locale (User/System)) : English/English | Czech/Czech
Free disk space on Miranda partition: 1346568 MBytes

Miranda NG Version: 0.95.3 alpha build #10240 x64
Build time: 20 Aug 2014 5:50:54
Profile: D:\Programs\Programs\Miranda NG\Profiles\Vulpix\Vulpix.dat
Profile size: 130220032 Bytes
Profile creation date: 26 Oct 2013 17:24:40
Language pack: No language pack installed
Service Mode: No

Active Plugins (48):
  AddContactPlus.dll v.0.9.9.2 [14 Feb 2014 6:47:44] - Add contact+
  AdvaImg.dll v.0.11.0.1 [2 Jan 2014 6:45:32] - Miranda image services
  AIM.dll v.0.11.0.1 [13 Aug 2014 5:42:20] - AIM protocol
  AssocMgr.dll v.0.1.1.0 [6 Aug 2014 5:48:32] - File association manager
  AuthState.dll v.0.0.2.1 [13 Aug 2014 5:43:04] - Authorization state
  AvatarHistory.dll v.0.1.0.1 [6 Aug 2014 5:48:38] - Avatar history
  AVS.dll v.0.95.3.1 [6 Aug 2014 5:43:28] - Avatar service
  ChangeKeyboardLayout.dll v.0.0.1.6 [13 Aug 2014 5:44:30] - Change keyboard layout
  ClientChangeNotify.dll v.0.1.1.3 [8 Aug 2014 5:48:56] - Client change notify
  Clist_modern.dll v.0.9.1.1 [6 Aug 2014 5:46:24] - Modern contact list
  Contacts.dll v.1.6.0.0 [6 Aug 2014 5:44:42] - Send/receive contacts
  CrashDumper.dll v.0.0.4.22 [1 Jul 2014 5:49:48] - Crash dumper
  CSList.dll v.0.1.1.0 [16 Mar 2014 0:38:54] - Custom status list
  DbEditorPP.dll v.3.2.0.1 [13 Aug 2014 5:45:58] - Database editor++
  Dbx_mmap.dll v.0.95.1.3 [20 Aug 2014 5:43:16] - Miranda NG mmap database driver
  FileAsMessage.dll v.0.0.2.4 [18 Jun 2014 12:59:54] - File as message
  Fingerprint.dll v.0.1.0.0 [13 Aug 2014 5:47:08] - Fingerprint NG
  Flags.dll v.0.1.0.4 [13 Aug 2014 5:43:12] - Country flags
  Folders.dll v.0.2.0.1 [5 Mar 2014 6:43:36] - Custom profile folders
  GTalkExt.dll v.0.1.1.0 [19 Aug 2014 5:49:20] - GTalk Extension
  HistoryPP.dll v.1.5.1.5 [22 Jul 2014 11:48:30] - History++
  ICQ.dll v.0.11.2.5 [17 Aug 2014 5:46:30] - IcqOscarJ protocol
  IEView.dll v.1.4.0.0 [6 Aug 2014 5:44:54] - IEView
  IgnoreState.dll v.0.0.3.0 [13 Aug 2014 5:43:26] - Ignore state
  Import.dll v.0.95.3.2 [17 Aug 2014 5:45:46] - Import contacts and messages
  Jabber.dll v.0.11.0.4 [19 Aug 2014 5:48:22] - Jabber protocol
  KeepStatus.dll v.0.9.0.2 [12 Jun 2014 5:43:16] - Keep status
  MSN.dll v.0.11.1.1 [19 Aug 2014 5:48:10] - MSN protocol
  mTextControl.dll v.0.8.0.1 [12 Jun 2014 5:42:58] - Miranda text control
  NewXstatusNotify.dll v.1.4.0.10 [6 Aug 2014 5:44:56] - xStatus notify
  Nudge.dll v.0.1.0.1 [6 Aug 2014 5:44:58] - Nudge
  OpenFolder.dll v.1.1.0.0 [15 Aug 2014 5:44:50] - Open Miranda folder
  PluginUpdater.dll v.0.1.2.2 [17 Aug 2014 5:50:02] - Plugin updater
  PManagerEx.dll v.0.0.0.5 [6 Aug 2014 5:47:42] - Miranda NG profile changer
  Restart.dll v.0.0.6.0 [9 Jul 2013 6:56:52] - Restart
  SeenPlugin.dll v.5.0.4.9 [15 Aug 2014 5:42:42] - Last seen
  SimpleStatusMsg.dll v.1.9.0.4 [15 Aug 2014 5:43:22] - Simple status message
  Skype.dll v.0.11.0.2 [7 Aug 2014 5:54:16] - Skype protocol
  SmileyAdd.dll v.0.2.3.17 [15 Aug 2014 5:42:52] - SmileyAdd
  StartPosition.dll v.0.2.0.3 [29 Jun 2014 5:50:44] - Start position
  Steam.dll v.0.11.0.1 [25 Jul 2014 18:22:48] - Steam protocol
  TabSRMM.dll v.3.4.0.3 [12 Aug 2014 16:52:58] - TabSRMM
  TopToolBar.dll v.0.8.0.3 [6 Aug 2014 5:46:08] - TopToolBar
  Tox.dll v.0.11.0.1 [20 Aug 2014 21:59:45] - Tox Protocol
  UInfoEx.dll v.0.8.5.0 [13 Aug 2014 5:44:06] - User info ext
  Variables.dll v.0.2.3.10 [6 Aug 2014 5:51:02] - Variables
  VersionInfo.dll v.1.5.0.3 [6 Aug 2014 5:47:16] - Version information
  Yahoo.dll v.0.11.0.1 [6 Aug 2014 5:44:08] - Yahoo protocol

Unloadable Plugins (1):
  DbChecker.dll v.0.0.0.0 [20 Aug 2014 5:47:04] - <unknown>


Protocols and Accounts:
-------------------------------------------------------------------------------
AIM                      1 - Enabled 0 - Disabled  Loaded
ICQ                      0 - Enabled 1 - Disabled  Loaded
JABBER                   0 - Enabled 2 - Disabled  Loaded
MSN                      0 - Enabled 2 - Disabled  Loaded
SKYPE                    0 - Enabled 2 - Disabled  Loaded
STEAM                    1 - Enabled 0 - Disabled  Loaded
TOX                      1 - Enabled 0 - Disabled  Loaded
YAHOO                    0 - Enabled 2 - Disabled  Loaded
IRC                      0 - Enabled 1 - Disabled  Not Loaded
MetaContacts             1 - Enabled 0 - Disabled  Loaded

Icon Packs:
-------------------------------------------------------------------------------
 Flags_icons.dll [6 Aug 2014 5:48:54]
 Fp_icons.dll [13 Aug 2014 5:47:02]
 HistoryPP_icons.dll [6 Aug 2014 5:47:50]
 Proto_AIM.dll [13 Aug 2014 5:44:36]
 Proto_conn_Skype.dll [3 Oct 2013 7:33:22]
 Proto_Facebook.dll [12 Aug 2014 5:50:44]
 Proto_GG.dll [6 Aug 2014 5:46:12]
 Proto_ICQ.dll [17 Aug 2014 5:48:24]
 Proto_IRC.dll [6 Aug 2014 5:46:12]
 Proto_Jabber.dll [19 Aug 2014 5:49:58]
 Proto_MetaContacts.dll [20 Aug 2014 5:49:54]
 Proto_MRA.dll [6 Aug 2014 5:46:10]
 Proto_mRadio.dll [8 Aug 2014 5:51:06]
 Proto_MSN.dll [19 Aug 2014 5:49:58]
 Proto_NewsAggregator.dll [15 Aug 2014 5:45:40]
 Proto_Omegle.dll [6 Aug 2014 5:47:50]
 Proto_Quotes.dll [13 Aug 2014 5:46:50]
 Proto_Skype.dll [7 Aug 2014 5:53:46]
 Proto_Steam.dll [25 Jul 2014 18:22:46]
 Proto_Tlen.dll [2 Jul 2014 5:45:44]
 Proto_Twitter.dll [6 Aug 2014 5:46:10]
 Proto_Weather.dll [17 Aug 2014 5:50:28]
 Proto_Yahoo.dll [6 Aug 2014 5:46:14]
 Proto_YAMN.dll [29 Jun 2014 5:50:24]
 SecureIM_icons.dll [13 Aug 2014 5:47:26]
 TabSRMM_icons.dll [12 Aug 2014 16:57:02]
 Toolbar_icons.dll [6 Aug 2014 5:46:16]
 UInfoEx_icons.dll [13 Aug 2014 5:44:16]
 Watrack_buttons.dll [6 Aug 2014 5:51:56]
 Watrack_icons.dll [6 Aug 2014 5:51:56]
 xStatus_ICQ.dll [17 Aug 2014 5:46:42]
 xStatus_Jabber.dll [19 Aug 2014 5:48:26]
 xStatus_MRA.dll [6 Aug 2014 5:45:48]

[close]

[Vulpix]

Quick update - I think it has to do with "convert old messages from unicode to utf8". I have tried running the check with "mark all events as read" and "aggressive" many times since the posting of this bug - and that issue has not manifested itself again.

[watcher]

I reproduced it once on test profile and then it stopped reproducing And ghazan needs fully repeatable bug to reproduce it on his machine.

[Vulpix]

Yeah it happened twice with my actual profile (I had backups the second time but not the first time) and then it stopped reproducing too. But it's definitely there.... unsane said he'll probably look into the issue a bit this weekend. Until then, I'm always making a backup before running the dbchecker.

I need to run it for pretty much every new tox contact I add - because for some reason, unless I do "mark all as read", the entire history of the contact always tries to load itself whenever I start a chat. Only happens with newly added contacts though.

[Vulpix]

I'm resurrecting this old topic because the issue occurred again. Again, it's the steam and tox protocol affected.

Luckily this time I have a before/after backup.

This is before:

Contact A, 19.10.2015 20:07:07:
God*

This is after:
Contact A, 19.10.2015 20:07:07:
肀賥肄蓣肴蓣肄胣肈郣肌


I can't find anything more relevant, just that it also corrupted some other chats that are from 21.05.2015 which is from so long ago... I have a backup from the 19.10.2015, 21:00, when the messages are still fine (even that one for 21.05.2015), so clearly the run of dbchecker afterwards messed up the history not just for recent but also for some older data.

I did not use the "convert" option. Just "mark as read" and "aggressive".

@unsane/robyer, could you please look into it? I obviously have both of my profiles, prior to and post-corruption, but they contain too much personal information to share and if I strip it then they will not be valuable.

There must be something wrong however because this doesn't happen with other protocols like jabber.

If there is some more info that you want me to provide, please let me know how I can get it. Maybe look at the particular message in db or something?

[Robyer]

Vulpix, hi, I've looked through all Steam proto and there are few parts that add events to DB. It is using his custom method "AddDBEvent()", which might be somehow wrong (but for first look it looks okay).

But it is questionable why unsane used this custom function on some places, where is usually used standard variant. One such place is method for receiving messages - CSteamProto::RecvMsg(), where he used "AddDBEvent" instead of classic "Proto_RecvMsg()" which is used in almost all other protocols (AIM, FB, GG, Jabber, MSN, Sametime, Twitter, VKontakte, WhatsApp, Yahoo). Own method is used in Tox too. My bet is this will be part of the problem.

We need response from unsane, why he not used standard method.
+ I'd like to have comment preferably from ghazan to clarify what could be wrong with that custom method (because it's used not only for receiving messages - so it still could break other events).


(... maybe there is just used some wrong flag for codepage?)

[unsane]

We need response from unsane, why he not used standard method.

Copy-paste driven development 

I removed custom AddDBEvent from tox and steam at r15606, r15606.

[Vulpix]

Interesting!

Is there a way to repair/recover the already-corrupted history entires? Or at least repair the current ones so that they don't become corrupted when I need to run the db check again? Right now I can't run it because it'll happen again.

Maybe there could be a debug log for the db checker which would tell us why it thinks those entries are wrong, and what exactly it does with them? (because prior to db check, they look fine in normal history. After db check I get chinese. Or japanese, whatever that is. Unicodese )

[Robyer]

Vulpix, try to create new profile and import everything from old profile there. But that probably won't help. Maybe try XML export via UserInfoEx? I'm not sure if there is any solution for the corrupted events...

[Vulpix]

Oh, no, no non on onononoono

I am  not importing my profile anywhere. It always loses some messages, some contacts and some other stuff; avatar history too.

I'm thinking I'll just use history export (txt), then history import (gonna switch to x86 miranda), and then re-bookmark all of my bookmarks manually. Seems like a hassle but last time I tried import, it just didn't work properly. The profile is maybe ten years old, from miranda IM times even. I'm getting a little nostalgic here but basically I once tried importing stuff and it just all broke...so no =P just no.

I plan on actually doing a full history export/import at some point, but it's going to be a problem because I've not yet solved the problem of a) avatar history, b) dummy contacts (since you can't just "create" a dummy contact :< ) and a few other things.

[Robyer]

Vulpix, did you tried new "full import" via Import.dll plugin? It should import everything.

Btw how would you imagine adding dummy contacts? How and with what result? :-)

[Robyer]

b) dummy contacts (since you can't just "create" a dummy contact :< )

I think you've needed something else, but I have improved Dummy protocol to allow searching and adding contacts. (And you can add contacts manually to other networks via AddContactPlus plugin)

Also users can send messages to Dummy contacts now. So Miranda could be raped into some kind of private notebook - user create new Dummy account, add some dummy contacts and then send messages to them (which will be saved to DB for their later use).