Database tools

[riki]

Many users, me included, have had several problems with database driver playing foul: total loss of data. Those of us without backups have had a terrible time. Still today, a "compact database" command destroyed my dat file.

• It would be nice to have some recovery tools, that can try to fix/recover a database file as it was with the old Miranda-NG. As it is now, when it's gone it's gone 
  
  
• A tool to delete duplicate messages would be nice, too. Here are some use cases that actually happened and still happen to me.
  
  
  • Facebook protocol often "misses" a message on the Miranda side, and even setting "automatically load messages when opening window" doesn't redownload it from FB server. The only way to get the message back is "download history", which retrieves all the messages (often in the thousands!) and leaves history full of duplicates.
    
    
  • Similar with old ICQ conversations that got partially lost in crashes long ago. Now I'm trying to use importTXT to recover from my old non-JSON text backups. I will have the same duplicate problem.
    
    

An alternative to Point 2, at least for Facebook, would be to be able to force "Download last N messages" so it downloads not the full history, but only the last N msgs. The duplicates in this case can be deleted by hand, if N is not too large. Do you think I should post this "download last N messages" request in the FB protocol subforum?

[Vulpix]

Tools to recover DB are present in the latest miranda, they should be in miranda base dir when you update.

usage:

Code: [Select]

mdbx_dump.exe -n -a -f 1.tmp "path/to/corrupted.dat"
mdbx_load.exe -n -f 1.tmp repaired.dat
del 1.tmp
I however tried them with no success when I reproduced an issue so your mileage may wary. (see post https://forum.miranda-ng.org/index.php?topic=5552.msg22340#msg22340 )

1) Unsane is working on some sqlite implementation. However since miranda's main core was not written for SQL-like requests, it is incredibly slow. Though Unsane improved this recently. I am yet to test this improvement however, but of course without some better core rewrite the slowness is going to persist. But then obviously, sqlite is a very common format with plenty of recovery tools.

2) to remove duplicate messages you can export the entire history once imported, then erase current history and import it with the "check for duplicates" mark checked. That should delete duplicates. Problem is duplicate testing is _still_ only based on timestamp so if you speak quickly you will also lose a potential nonduplicate message... so honestly...

I know how you feel, and I wish we'd never abandoned mmap...

[riki]

I've read the thread you linked. My oh my... luckily, you are in the "backup everything every day" club. But stressful, eh?

By the way, autobackups aren't functional, either. Apparently, it's still impossible to duplicate an open database - as I'm sure you know well. The only way I found that works is closing down Miranda and copy the file via OS, then reopen Miranda. Just an additional teaspoon of stress.

2) to remove duplicate messages you can export the entire history once imported, then erase current history and import it with the "check for duplicates" mark checked. That should delete duplicates. Problem is duplicate testing is _still_ only based on timestamp so if you speak quickly you will also lose a potential nonduplicate message... so honestly...

I thought the new db driver had an index for objects that would make it possible to check for uniqueness without resorting to unreliable timestamp tests? 

[Vulpix]

Autobackups (via the autobackup plugin) don't copy the file; they perform a copy of the database into a new profile, more or less. Autobackups work fine for me (now), but they had failed me previously (same as they are failing you now, I reckon). I think the reason they don't work for you is because they basically do an export of your profile. Since you use 32bit miranda and your profile may or may not be corrupted, the export fails. Very fatal, I know; especially because it's a feature supposed to be a failsafe against losing your data. If the failsafe is not safe, am I even real?

You can sort of see if this is the case by examining the size of the exported autobackups. If it fluctuates weirdly (like 100mb 99mb 101mb 100mb 100mb 101mb 99mb) etc.. basically not really increasing as you would expect but just being all over the place, it means the export is not working properly).

So yeah I backup manually. And JSONexports via msgexport, they have saved my skin.

As for uniqueness, yes - however, currently the code is not there for facebook, and was only recently (yesterday? ) added to Discord. In other words, the DB supports it, and recently an method/api was added that lets plugins use it, but the support for it has yet to be coded for Facebook and other "retrieved from server" kinds of protocols.