Fatal errors that cannot be remedied with Repair collection

SUMMARY

If you experience major problems with your data in SuperMemo, and your collection repair fails, you may experience a rare bug that affects probably all versions of SuperMemo from version 8.0 to version 18.05.

If you recover.txt report contains the following error "Stray files at filespace slot", the recovery may trip on the bug and fail.

To see if this is the case, you can run repairs with this debug version of SuperMemo compiled specifically to recover from the bug problem:

https://supermemo.org/users/merin/sm18.zip

If you do not experience the problem above, do not download. To substitute SuperMemo 18 for the above version, replace sm18.exe with the file included in the ZIP file.

Note: An otherwise innocent error "Missing filespace slot" may accompany "stray files" but should not be a sole cause for worry.

Read the texts below only to find the context in which the bug was discovered:

Problem

anonymous wrote:

I like to stay anonymous. I've been using Supermemo for a year now and enjoying it very much. I've added a couple of really big books. This took so long that windows gave the "supermemo is not responding do you want to close it?" message. Due to me not having any patience I forced a closing of supermemo, not realizing the potential damage this could inflict on my collection. Now recently I've been receiving fatal errors. According this I should have already emailed you:

https://super-memory.com/archive/help2002/errors.htm#Fatal%20errors

When I receive these I make a backup and run repair collection. This usually fixes it. However this morning I got a fatal error and after running repair collection my data was corrupted. The knowledge tree got completely reshuffled around concepts (I never use concepts), priorities changed, extracts I never made, weird answers to flashcard (things such as "pp"). In other words: completely kaput. I tried opening the backup I made this morning, which gave a lot of registry errors and then similar madness. The good news is that there's an automatic backup made on an external harddrive 1-2 weeks ago, however I'm scared that if I open it it will also get corrupted. Is this still salvageable?

User Followup

I've merged the two collections but it appears this only added the data of the uncorrupted backup to the corrupted collection, so I'm not sure what the benefit was. And even if it worked it would not have addressed the core issue: regular fatal errors, which means the collection could recorrupt at any moment.

Using Merge

If you use merge, always transfer data from the corrupted collection (if salvagable) to the uncorrupted one. Moreover, you should rather only transfer newly added material (which can be sifted out with the help of subsets)

First comments

Upon inspecting your collections, here are some quick comments:

• indeed, some files cause problems (23MB books are rarely tested in SuperMemo)
• Repair collection trips on one of your collections, possibly due to data corruption. Such an inability to cope with recovery will be studied urgently
• your undamaged collection might be used in the meantime, and, in case of trouble, new data might later be moved back to the backup

Unusual find that might trigger a bug

There seems to be a bug in SuperMemo, which can be triggered by having an mp3 and htm file stored at the same filespace slot.

In your case you could see:

supermemo_backup\elements\3\5\18\28690.HTM (22 November 2021)

and

supermemo_backup\elements\3\5\18\28690.mp3 (21 March 2022)

Do you have an idea how those two files ended up in the same slot? This looks like a synchronization problem. You might have combined files from an older and newer versions of the same collection. This can also happen when coping one collection over the other.

Whatever the reason, there should be two things done:

1. bug trigger: you need to see if duplicated slot is not a result of erroneous backup/synchronization procedure
2. but itself: the bug triggered by the double slot will be fixed (it is about using multiple file searches asynchronously without sufficient isolation of separate processes)

User observation

You say your hardware is responsible for automatic synchronization. If so, it is important to avoid synchronization while using SuperMemo! This could be the exact source of problem. You synchronize file A, and a millisecond later the file gets deleted and replaced with file B. If file B gets cloned, you will end up with files A and B in the same slot on the backup drive.

In your case, filespace slot duplication lead to uncovering a bug that might have been in SuperMemo for years unnoticed as it shows up only in case of said duplicate.

Solution?

This is probably what you should do at the moment:

• continue learning with the backup collection
• download SM18 with a bug fix (it will be posted on this page asap)
• backup again
• run File : Repair collection
• make manual fixes to items with errors (insignificant, e.g. empty items, items with gaps in repetition history)

Another user's story

Summary: I was in a very similar situation. Luckily I had some surviving (but also contaminated) backup files, and now I can safely enjoy SM again thanks to this documentation. It reminded me of one fundamental approach: thinking about my actions, not SM functions, that may have caused the problem (e.g. collection destroy).

Before the official bugfix release, I came up with two options:

• option 1: "neater" but "slower" fix
• option 2: "messier" but "quicker" fix (e.g. trying to fix on my own, allowing reshuffled knowledge-tree(e.g. three links between topic-extract-extract-item were broken and it was scattered here and there), or troubles that I may have asked for, etc)

It took me a little while, since my collection was literally "completely kaput"; If I had a somewhat small collection, I honestly wouldn't hesitate to opt for "neat and quick" fix. Anyway, I just wanted my collection to breathe. I chose option 2 with learning efficiency in mind. In addition, thinking of efficiency of another kind, I took a walk and thought about the potential causes, instead of scrutinizing the entire collection.

Here's what I was thinking:

• Not many, but various program script components were used (e.g. some .py/.pyw files were being executed, saved, or overwritten during my incremental creativity slot. Also, this is speculation, I think I may have been working with some relevant files right before or after running SM backup. Or something like that.)
• There were some PDF files that were a bit thick (e.g. a bunch of mathematics textbooks that were difficult to replace with Wikipedia)
• Security program issues (e.g. a good few security related programs are required for doing internet banking or accessing specific bank websites on a PC, at least in my country. But there are known or unknown annoying aspects such as: blocking potentially harmful factors, slowing down PC, blocking mstsc, etc)
• Ignorance of settings as to "concepts"

Perhaps parts of the list above have nothing to do with the exact cause, but I treated the list as material for: rethinking true priorities, revising my strategy, getting inspired, etc (in conjunction with this documentation):

• I deleted all .py/.pyw program script components. Unlike other useful cases (e.g. PDF, e-library links), it was actually more necessary to open files as editor mode than to run the program; it would have been more convenient to just save the file path in text format, or create a shortcut icon on the desktop, etc. Meanwhile, I also rediscovered the value of "merging after working with multiple collections" as to incremental creativity, rather than always staying with one big collection.
• I said goodbye to almost all math PDF textbooks; I actually dropped out of mathematics graduate school (getting a little bit serious about: my mental health, my mathematics-potential in a long-term perspective, etc)
• For a while, I decided to do internet banking via smartphone (e.g. as simple as: just a few touches / one biometric authentication), and not via PC (e.g. complex and heavy procedures). In fact that way is a lot more convenient, unless I need to do some serious work on specific bank websites on a regular basis, etc.
• I looked up some help documents as to "concepts" (e.g. "avoid using the #1 element as the root")

That's not all, but the whole recovery process didn't take much time (e.g. within a day). The one that took the longest was to choose option 2 (e.g. almost half of the day incl. sleep). As of now, I no longer receive fatal error messages (e.g. receiving copy/backup/repair related errors when trying to copy/backup/repair)

Of course, I was lucky enough. I was very lucky that I had frequent backups (as in other fields). If it wasn't for this... what a disaster, let alone option 2, or even option 1! Not only that, I was also lucky to have taken a rather general approach when it comes to templates within SM. For example, I've rarely produced a variety of complex templates, except in a few special cases. I also didn't really care much about adding pictures unless it was really necessary. So, during the recovery process, I was able to apply the default Article/Item template to topics/items in a lump (though may be subject to resizing). Meanwhile, elements that once used special templates or contained pictures will be incrementally modified according to priorities, needs, etc.

Well, what I have done here may seem like a lot of sacrifice on the surface (e.g. chaotic knowledge tree), but in reality the quality of learning (e.g. but still coherent memory) is hardly lost (warning: the nuance I'm trying to give is not something ideological, but has more to do with the well-known idea that the outer shell and the inner essence may or may not be consistent. And indeed, ideas like "quality of learning" or "learning efficiency" here can be misunderstood without a proper SM context, although it's not impossible for one for all times and places to get very similar ideas in an unexpected way, just like the story of Newton and Leibniz)! (cf. "knowledge tree is not essential for your success in learning") Looking back, I unwittingly experienced a similar (but not identical) version of the "five stages of grief" for a week as to my collection, including writing this story. Now, metaphorically, in terms of thermodynamics, ΔG<0. Hope you too can salvage your collection! Bye-bye.

P.S. I feel sorry for not being able to explain the situation for a perfect reproduction of the bug. I was so immersed in IC that I can't remember exactly what was going on around that time. After working all day, I did a backup and repair before going to bed, but an error occurred. I thought it would be fixed if I repaired it, but the error gradually increased by hundreds or thousands. So I thought if I repaired it again, it would be fixed, but it got weirder, and some elements gradually became blank, and my knowledge tree became weird too. Almost at the same time, I think I drank a beer and did a few things and then went to bed. Looking back, it seems that not only the bugs described in this article were mixed, but also my ignorance. Anyway, it was an important opportunity to realize the importance of backup again.

Comment on the original case

It seems the original collection does not require much salvaging. It was fixed easily with the corrected version of SuperMemo (dealing with the duplicated slot). The user had a fresh backup, which is always the best remedy to fall back on