Duplicate detection not working properly in 4.5.2+

[Hemloc]

Since upgrading to 4.5.2 on Windows 10, the duplicate detection no longer works properly. It is indicating an anime episode is a duplicate, when it is not.

E.g. After downloading "[SubsPlease] Kakkou no Iinazuke S2 - 10 (1080p) [6A7D01C9]", if I try download "[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF]" it rejects the download as a duplicate:

2025-09-10 18:02:03,715::INFO::[notifier:165] Sending notification: NZB added to queue - [SubsPlease] Kakkou no Iinazuke S2 - 10 (1080p) [6A7D01C9].nzb (type=download, job_cat=*)
...
2025-09-10 18:02:50,591::INFO::[notifier:165] Sending notification: Warning - Ignoring duplicate NZB "[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF]" (type=warning, job_cat=None)

I then permanently deleted 10 from the archive (this is very poorly logged as it only indicates the job removed from history, not the filename which would be much more useful) :
2025-09-10 18:13:32,740::INFO::[database:218] [N/A] Removing job SABnzbd_nzo_qo42i7sb from history

Then added 09 again, and suddenly it is no longer a duplicate:

2025-09-10 18:14:32,815::INFO::[notifier:165] Sending notification: NZB added to queue - [SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF].nzb (type=download, job_cat=*)

Then trying to download 10 again we get:

2025-09-10 18:15:11,522::WARNING::[misc:103] Ignoring duplicate NZB "[SubsPlease] Kakkou no Iinazuke S2 - 10 (1080p) [6A7D01C9]"


Unfortunately constantly asking users to visit the Reddit "Restart Sonarr created DL, flagged/blocked duplicate by SABnzd" every time someone has problems with duplicate entries is really not the correct way of handling this any more. Users have been complaining for years about this, so a simple "Download Anyway" button next to detected duplicates would make life much easier for everyone.

[sander]

Hi Hemloc, welcome to the forum!

> Unfortunately constantly asking users to visit the Reddit "Restart Sonarr created DL, flagged/blocked duplicate by SABnzd"

Who / where?

[Hemloc]

Ah, seems that part is something I saw when doing general checks for how to handle duplicates when searching the internet, and obviously reddit is a common place for having questions asked and answered. In typical fashion, on trying to find them again on the internet, I don't get much! So the only place on the forum I found this is under "flagged/blocked duplicate by SABnzdb", and oddly enough, by you So I guess apologies are in order, saying something that doesn't actually appear to be true, so sorry about that.

Still, on checking on how to retry downloads there does not seem to be a simple answer, with the only options being "disable duplicate checking" as per the "WARNING Ignoring duplicate NZB [FILENAME.NZB]" topic from 9 years ago, and the above mentioned topic from 8 months ago (so no change after so many years ). Or the option of having to go find the original filename in the archive, delete it. before having to then manually add the nzb file again.

There does not seem to be a Suggestion topic on this forum, but simply being able to click a "Download Duplicate Anyway" type button would make all these problems of duplicate filenames go away. Just a thought

[sander]

I'm confused:

> Since upgrading to 4.5.2 on Windows 10, the duplicate detection no longer works properly

versus

> Users have been complaining for years about this

So which one is it?

Easy check: use SABnzbd 4.5.1 and report if that works for you.

[Hemloc]

This forum is for bugs, so the duplicate detection problem.

I tried 4.2.3 with the same problem, so this bug appears to have actually been there for quite some time.

[safihre]

I'm currently away on holiday, so can't check directly.
But if you enable Debug logging in the Status window and then after it happens again you can check the log.
It will log how it detected the duplicate, based on what show and season/episode detection.

Maybe this release just uses a format that isn't detected. Which kinda makes sense, S2 - 10 is not something I've ever seen before. Usually it's S02E10 or even S02x10.

[Hemloc]

I have realised why only now has this come up. It is because I have increased the number of days History will keep, it was previously 3 days, now it is 14, and since the episodes are released weekly, that is why they are suddenly being seen as duplicates.

I enabled debugging, but the logging doesn't show me anything about a season or show:

2025-09-11 19:09:23,976::INFO::[notifier:165] Sending notification: SABnzbd - SABnzbd 4.5.3 started (type=startup, job_cat=None)
2025-09-11 19:10:27,611::INFO::[nzbparser:85] Attempting to add [SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF].nzb.gz
2025-09-11 19:10:27,612::INFO::[filesystem:726] Creating directories: \\?\D:\# Newsleecher Temp\[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF]
2025-09-11 19:10:27,612::INFO::[filesystem:726] Creating directories: \\?\D:\# Newsleecher Temp\[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF]\__ADMIN__
2025-09-11 19:10:27,612::INFO::[filesystem:1218] Saving \\?\D:\# Newsleecher Temp\[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF]\__ADMIN__\[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF].nzb.gz
2025-09-11 19:10:27,963::DEBUG::[interface:144] Request GET /api from 127.0.0.1 [Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36] {'mode': 'queue', 'start': '0', 'limit': '20', 'output': 'json', 'apikey': '7b57070a1646496c80d28b144af38959', '_': '1757610571081'}
2025-09-11 19:10:27,964::DEBUG::[interface:144] Request GET /api from 127.0.0.1 [Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36] {'mode': 'history', 'failed_only': '0', 'start': '0', 'limit': '20', 'archive': '0', 'last_history_update': '1', 'output': 'json', 'apikey': '7b57070a1646496c80d28b144af38959', '_': '1757610571082'}
2025-09-11 19:10:28,404::DEBUG::[nzbparser:402] NZB file meta-data = {'x-generator': ['Nyuu [https://animetosho.org/app/nyuu]'], 'title': ['[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF].mkv'], 'x-info-url': ['https://animetosho.org/view/2015753']}
2025-09-11 19:10:28,405::INFO::[nzbstuff:978] File [SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF].par2 added to queue
2025-09-11 19:10:28,425::DEBUG::[filesystem:1109] [sabnzbd\nzbstuff.py.__init__] Saving data for SABnzbd_nzf_uiqebjq9 in \\?\D:\# Newsleecher Temp\[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF]\__ADMIN__
2025-09-11 19:10:28,425::INFO::[nzbstuff:978] File [SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF].vol000+001.par2 added to queue
2025-09-11 19:10:28,430::DEBUG::[filesystem:1109] [sabnzbd\nzbstuff.py.__init__] Saving data for SABnzbd_nzf_ho49rxg8 in \\?\D:\# Newsleecher Temp\[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF]\__ADMIN__
2025-09-11 19:10:28,431::INFO::[nzbstuff:978] File [SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF].mkv added to queue
2025-09-11 19:10:28,431::DEBUG::[filesystem:1109] [sabnzbd\nzbstuff.py.__init__] Saving data for SABnzbd_nzf_erpzim7n in \\?\D:\# Newsleecher Temp\[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF]\__ADMIN__
2025-09-11 19:10:28,432::INFO::[nzbstuff:978] File [SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF].vol001+002.par2 added to queue
2025-09-11 19:10:28,432::DEBUG::[filesystem:1109] [sabnzbd\nzbstuff.py.__init__] Saving data for SABnzbd_nzf_7dhqcg73 in \\?\D:\# Newsleecher Temp\[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF]\__ADMIN__
2025-09-11 19:10:28,432::INFO::[nzbstuff:978] File [SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF].vol003+004.par2 added to queue
2025-09-11 19:10:28,432::DEBUG::[filesystem:1109] [sabnzbd\nzbstuff.py.__init__] Saving data for SABnzbd_nzf_lb_tzlql in \\?\D:\# Newsleecher Temp\[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF]\__ADMIN__
2025-09-11 19:10:28,433::INFO::[nzbstuff:978] File [SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF].vol007+008.par2 added to queue
2025-09-11 19:10:28,433::DEBUG::[filesystem:1109] [sabnzbd\nzbstuff.py.__init__] Saving data for SABnzbd_nzf_of9_2rw3 in \\?\D:\# Newsleecher Temp\[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF]\__ADMIN__
2025-09-11 19:10:28,433::INFO::[nzbstuff:978] File [SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF].vol015+016.par2 added to queue
2025-09-11 19:10:28,434::DEBUG::[filesystem:1109] [sabnzbd\nzbstuff.py.__init__] Saving data for SABnzbd_nzf_u1pfoe05 in \\?\D:\# Newsleecher Temp\[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF]\__ADMIN__
2025-09-11 19:10:28,434::INFO::[nzbstuff:978] File [SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF].vol031+032.par2 added to queue
2025-09-11 19:10:28,434::DEBUG::[filesystem:1109] [sabnzbd\nzbstuff.py.__init__] Saving data for SABnzbd_nzf_fo9wch9_ in \\?\D:\# Newsleecher Temp\[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF]\__ADMIN__
2025-09-11 19:10:28,434::INFO::[nzbstuff:978] File [SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF].vol063+039.par2 added to queue
2025-09-11 19:10:28,435::DEBUG::[nzbstuff:1987] Duplicate checking NZB [SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF] (md5sum=941d5059147b601221d184bf1a35e084)
2025-09-11 19:10:28,435::DEBUG::[nzbstuff:1990] Duplicate in history: True
2025-09-11 19:10:28,435::DEBUG::[nzbstuff:1993] Duplicate in queue: False
2025-09-11 19:10:28,435::DEBUG::[misc:278] Parsing category * to attributes: pp=3 script=Default prio=0
2025-09-11 19:10:28,436::INFO::[notifier:165] Sending notification: Warning - Ignoring duplicate NZB "[SubsPlease] Kakkou no Iinazuke S2 - 09 (1080p) [5C6EC1AF]" (type=warning, job_cat=None)

Yes, this is not a normal TV series, this is an Anime series, and they often have different formats for their filenames compared to normal TV series. If you go to animetosho.org you will see a lot of the filenames supplied for anime. Some will have the S0xE0y format, but many will just have the title, followed by a hyphen, then the episode number, and if this is season 2 or more, put Sx before the hyphen (such as the example I gave above).

There are simply too many formats for filenames, which is why I also suggested that the program have an option to allow a user to download anyway a file it has thought was a duplicate.

[safihre]

No I meant the log for when it's falsely detected as duplicate. This looks like the first add.

[Hemloc]

Sorry about that, I added the wrong nzb file, I think this is what you were looking for:

2025-09-11 21:07:37,357::DEBUG::[nzbstuff:1987] Duplicate checking NZB [SubsPlease] Kakkou no Iinazuke S2 - 10 (1080p) [6A7D01C9] (md5sum=14c03ea37a0f7f4919220088100ac64a)
2025-09-11 21:07:37,358::DEBUG::[nzbstuff:1990] Duplicate in history: False
2025-09-11 21:07:37,358::DEBUG::[nzbstuff:1993] Duplicate in queue: False
2025-09-11 21:07:37,366::DEBUG::[sorting:657] Initial guess for [SubsPlease] Kakkou no Iinazuke S2 - 10 (1080p) [6A7D01C9] is MatchesDict({'release_group': 'SubsPlease', 'title': 'Kakkou no Iinazuke', 'season': 2, 'episode_title': '10', 'screen_size': '1080p', 'crc32': '6A7D01C9', 'type': 'episode'})
2025-09-11 21:07:37,366::DEBUG::[nzbstuff:2003] Smart duplicate checking ([SubsPlease] Kakkou no Iinazuke S2 - 10 (1080p) [6A7D01C9]): kakkou no iinazuke/2/
2025-09-11 21:07:37,366::DEBUG::[nzbstuff:2006] Duplicate in history: True
2025-09-11 21:07:37,366::DEBUG::[nzbstuff:2009] Duplicate in queue: False

[jcfp]

The nonstandard way of denoting the season/episode combo isn't helping:

Code: Select all

>>> import guessit
>>> guessit.guessit("[SubsPlease] Kakkou no Iinazuke S2 - 10 (1080p) [6A7D01C9]")
MatchesDict([('release_group', 'SubsPlease'), ('title', 'Kakkou no Iinazuke'), ('season', 2), ('episode_title', '10'), ('screen_size', '1080p'), ('crc32', '6A7D01C9'), ('type', 'episode')])

As you can see, that "S2 - 10" notation causes the episode number to go missing from the result, leaving sabnzbd with just season 2 and the "10" part taken for an episode title instead. The common notations of "S2E10", "S2x10", as well as many variations of those all are correctly detected as season 2 episode 10. The hyphen in this context is typically used to indicate a range, and unsurprisingly "S2-10" (with the spaces removed) would be detected as seasons 2 through 10.

According to your latest log exerpt, sabnzbd considers this a dupe because there's already a season 2 entry in the history database, which looks like a sensible way to handle this (considering season packs are a thing). Fixing the season/episode notation on this nzb looks like the obvious fix.

[Hemloc]

I presume that means fixing GuessIt since we cannot rename the thousands of existing anime filenames.
There should still be an easy way to download files found to be "duplicate". Just like the function name says, it is guesswork, and when it guesses wrong, the user should be able to say download it anyway without having to do a bunch of manual other stuff just to get the file.

[jcfp]

Guessit is pretty accurate with anything that even remotely resembles "scene naming". The naming scheme for these jobs is just weird enough to throw a wrench into the works. If this particular scheme is common for anime, you could also opt for a pre-queue script to make a minor edit to the season/episode notation in the job names rather than hoping for guessit to pick up on this.

There's multiple options for how to handle duplicate jobs, including simply pausing the job which makes it trivial to download anyway.

[Hemloc]

Agreed, or like simply allowing the user to download the file anyway, instead of simply rejecting something and therefore not giving the user any choice but have to find some other way of downloading the file. Remember, we don't just download TV episodes or movies on usenet, we download anything nowadays, so there is no way GuessIt will ever 100% accurate, so it should cater for this fact.

In any case, GuessIt is wrong, and this is the standard for anime scene releases, so I would say it should be updated to cater for anime standards.

[OneCD]

As mentioned by @jcfp, if your SABnzbd duplicate detection action is ‘pause’, you can manually resume the download.

[Hemloc]

Ah, I did not realise there was such an option (there are a lot of them), and yes, and it does exactly what I have been suggesting regarding the handling of Smart Duplicates, so thank you. And apologies for ranting on about something that was already there!