Algorithm SM-17 vs. older SuperMemos

From SuperMemopedia
Jump to navigation Jump to search

Question

Did you compare Algorithm SM-17 with older algorithms like SM-2, Anki, SM-8, etc.? Do you have specific numbers?

Hints

  • DSR model makes it possible to simulate and compare all imaginable review scenarios
  • validity of comparisons depends on the validity of the model, which is still work in progress
  • some simple approximation formulas can be used to yield rough comparisons without simulating actual algorithms
  • older SuperMemos do not consider the R dimension and they will fail each time review departs from the optimum determined by the forgetting index
  • even Algorithm SM-15 which approximates the impact of the spacing effect with a heuristic shows its weakness here: Is Algorithm SM-17 much better than Algorithm SM-15?

Answer

Yes, we did basic comparisons with excellent news for Algorithm SM-17. We will publish relevant numbers in the future.

Technical

Approximation procedures used in Algorithm SM-17 make it easy to compare various stability increase functions with actual stability data collected in the learning process. Some examples below make it possible to make rough judgements:

Algorithm SM-2

The following formula may approximate the stability increase function in Algorithm SM-2:

 SInc=1.2+(1-Diff)*2.3

where

  • SInc - stability increase
  • Diff - item difficulty (as in Algorithm SM-17)(which does not translate directly to E-factor used in SuperMemo 2)

This particular formula produces matrix deviation of 72% on a large dataset.

The obvious weaknesses of Algorithm SM-2 are: (1) lack of the retrievability dimension (e.g. poor response to review rescheduling), and (2) lack of power decline in stability increase (SInc) with increase in stability (S). However, those will not show if the user stick well to the prescribed repetition schedule. The increase in forgetting with longer intervals will be well compensated with reduced workload and related increase in the speed of learning (at the cost of retention). In other words, despite a large deviation form the true stability increase function, the algorithm used to perform well and make many users happy with the rapid increase in knowledge.

Actual SM-2 data

Feb 22, 2018: We have finally added a simulation of Algorithm SM-2 to SuperMemo for Windows and have come up with the average least squares metric of 53.5685% (for Algorithm SM-2). For comparison, Algorithm SM-17 results in 37.1202% (a million repetitions dataset). This may not sound impressive, however, for shorter intervals, the load of repetitions might easily be 2-10x greater assuming no delays (i.e. executing repetitions as prescribed). Back in 1989, we could see that even Algorithm SM-5 would reduce repetition loads twice as fast as SM-2.

For comparison, Algorithm SM-17 rarely goes 5pp above Algorithm SM-15 in R-Metric measure, and even small gains have a significant impact on workload (esp. at shorter intervals).

About the least squares metric: if predicted R is 70% and the grade is Success, the deviation is 30%. This value is squared and squares are averaged over all repetitions. The minimum metric for perfect predictions for a perfect string of repetitions at R=0.9 is 18%. Metric 0% is not possible (in theory, perfect prediction of a perfect string of R=1.00 might yield such a result).

Retrievability prediction in SM-2 can be obtained from:

SM2R:=Exp(-MDC*Int/SM2Int)

where:

  • SM2R - retrievability prediction for Algorithm SM-2
  • MDC - memory decay constant
  • SM2Int - interval proposed by Algorithm SM-2
  • Int - actual interval used in SuperMemo

See also:

Algorithm SM-8

The following formula may approximate the stability increase function in Algorithm SM-8:

 SInc=1.1+f*(1-Diff)*power(S,d)

where:

  • SInc - stability increase
  • Diff - item difficulty (as in Algorithm SM-17)(which does not translate directly to A-factor used in SuperMemo 8)
  • f - scaling constant
  • d - stability decay constant

This particular formula produces matrix deviation of 44% on a large dataset.

Algorithm SM-8 also misses the retrievability dimension. The problem was remedied with spacing effect heuristics only in Algorithm SM-11. However, O-Factors in Algorithm SM-8 show power decline with the increase in the repetition number, which is a major step forward compared with Algorithm SM-2, where E-factors could only be reduced by increase the presumed item difficulty.