bump to 0.6

README.md

@@ -18,33 +18,34 @@ hands-free as possible, TubeSync has gradual retrying of failures with back-off
 so media which fails to download will be retried for an extended period making it,
 hopefully, quite reliable.
 
+
 # Latest container image
 
 ```yaml
-ghcr.io/meeb/tubesync:v0.5
+ghcr.io/meeb/tubesync:v0.6
 ```
 
 # Screenshots
 
 ### Dashboard
 
-![TubeSync Dashboard](https://github.com/meeb/tubesync/blob/main/docs/dashboard.png?raw=true)
+![TubeSync Dashboard](https://github.com/meeb/tubesync/blob/main/docs/dashboard-v0.5.png?raw=true)
 
 ### Sources overview
 
-![TubeSync sources overview](https://github.com/meeb/tubesync/blob/main/docs/sources.png?raw=true)
+![TubeSync sources overview](https://github.com/meeb/tubesync/blob/main/docs/sources-v0.5.png?raw=true)
 
 ### Source details
 
-![TubeSync source details](https://github.com/meeb/tubesync/blob/main/docs/source.png?raw=true)
+![TubeSync source details](https://github.com/meeb/tubesync/blob/main/docs/source-v0.5.png?raw=true)
 
 ### Media overview
 
-![TubeSync media overview](https://github.com/meeb/tubesync/blob/main/docs/media.png?raw=true)
+![TubeSync media overview](https://github.com/meeb/tubesync/blob/main/docs/media-v0.5.png?raw=true)
 
 ### Media details
 
-![TubeSync media-details](https://github.com/meeb/tubesync/blob/main/docs/media-item.png?raw=true)
+![TubeSync media-details](https://github.com/meeb/tubesync/blob/main/docs/media-item-v0.5.png?raw=true)
 
 
 # Requirements
@@ -98,7 +99,7 @@ Finally, download and run the container:
 
 ```bash
 # Pull a versioned image
-$ docker pull ghcr.io/meeb/tubesync:v0.5
+$ docker pull ghcr.io/meeb/tubesync:v0.6
 # Start the container using your user ID and group ID
 $ docker run \
   -d \
@@ -109,7 +110,7 @@ $ docker run \
   -v /some/directory/tubesync-config:/config \
   -v /some/directory/tubesync-downloads:/downloads \
   -p 4848:4848 \
-  ghcr.io/meeb/tubesync:v0.5
+  ghcr.io/meeb/tubesync:v0.6
 ```
 
 Once running, open `http://localhost:4848` in your browser and you should see the
@@ -121,7 +122,7 @@ Alternatively, for Docker Compose, you can use something like:
 
 ```yaml
   tubesync:
-    image: ghcr.io/meeb/tubesync:v0.5
+    image: ghcr.io/meeb/tubesync:v0.6
     container_name: tubesync
     restart: unless-stopped
     ports:
@@ -196,10 +197,10 @@ $ docker logs --follow tubesync
 
 ### 1. Index frequency
 
-It's a good idea to add sources with as low an index frequency as possible. This is the
-duration between indexes of the source. An index is when TubeSync checks to see
+It's a good idea to add sources with as long of an index frequency as possible. This is
+the duration between indexes of the source. An index is when TubeSync checks to see
 what videos available on a channel or playlist to find new media. Try and keep this as
-long as possible, 24 hours if possible.
+long as possible, up to 24 hours.
 
 
 ### 2. Indexing massive channels
@@ -222,7 +223,7 @@ automatically.
 
 ### Does TubeSync support any other video platforms?
 
-At the moment, no. This is a first release. The library TubeSync uses that does most
+At the moment, no. This is a pre-release. The library TubeSync uses that does most
 of the downloading work, `youtube-dl`, supports many hundreds of video sources so it's
 likely more will be added to TubeSync if there is demand for it.
 
@@ -236,7 +237,7 @@ your install is doing check the container logs.
 ### Are there alerts when a download is complete?
 
 No, this feature is best served by existing services such as the execelent
-[tautulli](https://tautulli.com/) which can monitor your Plex server and send alerts
+[Tautulli](https://tautulli.com/) which can monitor your Plex server and send alerts
 that way.
 
 ### There's errors in my "tasks" tab!
@@ -293,12 +294,13 @@ Just `amd64` for the moment. Others may be made available if there is demand.
 # Advanced configuration
 
 There are a number of other environment variables you can set. These are, mostly,
-**NOT** required to be set in the default container installation, they are mostly
+**NOT** required to be set in the default container installation, they are really only
 useful if you are manually installing TubeSync in some other environment. These are:
 
 | Name                     | What                                  | Example                            |
-| ----------------- | ------------------------------------- | ---------------------------------- |
-| DJANGO_SECRET_KEY | Django secret key                     | YJySXnQLB7UVZw2dXKDWxI5lEZaImK6l   |
+| ------------------------ | ------------------------------------- | ---------------------------------- |
+| DJANGO_SECRET_KEY        | Django's SECRET_KEY                   | YJySXnQLB7UVZw2dXKDWxI5lEZaImK6l   |
+| DJANGO_FORCE_SCRIPT_NAME | Django's FORCE_SCRIPT_NAME            | /somepath                          |
 | TUBESYNC_DEBUG           | Enable debugging                      | True                               |
 | TUBESYNC_HOSTS           | Django's ALLOWED_HOSTS                | tubesync.example.com,otherhost.com |
 | GUNICORN_WORKERS         | Number of gunicorn workers to spawn   | 3                                  |
@@ -309,7 +311,7 @@ useful if you are manually installing TubeSync in some other environment. These
 # Manual, non-containerised, installation
 
 As a relatively normal Django app you can run TubeSync without the container. Beyond
-the following rough guide you are on your own and should be knowledgeable about
+following this rough guide you are on your own and should be knowledgeable about
 installing and running WSGI-based Python web applications before attempting this.
 
 1. Clone or download this repo

tubesync/sync/migrations/0005_auto_20201219_0312.py

@@ -0,0 +1,18 @@
+# Generated by Django 3.1.4 on 2020-12-19 03:12
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('sync', '0004_source_media_format'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='source',
+            name='source_type',
+            field=models.CharField(choices=[('c', 'YouTube channel'), ('i', 'YouTube channel by ID'), ('p', 'YouTube playlist')], db_index=True, default='c', help_text='Source type', max_length=1, verbose_name='source type'),
+        ),
+    ]

tubesync/sync/migrations/0006_source_write_nfo.py

@@ -0,0 +1,18 @@
+# Generated by Django 3.1.4 on 2020-12-19 03:12
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('sync', '0005_auto_20201219_0312'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='source',
+            name='write_nfo',
+            field=models.BooleanField(default=False, help_text='Write an NFO file with the media, these may be detected and used by some media servers', verbose_name='write nfo'),
+        ),
+    ]

tubesync/sync/migrations/0007_auto_20201219_0645.py

@@ -0,0 +1,18 @@
+# Generated by Django 3.1.4 on 2020-12-19 06:45
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('sync', '0006_source_write_nfo'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='source',
+            name='write_nfo',
+            field=models.BooleanField(default=False, help_text='Write an NFO file in XML with the media info, these may be detected and used by some media servers', verbose_name='write nfo'),
+        ),
+    ]

tubesync/sync/migrations/0008_source_download_cap.py

@@ -0,0 +1,18 @@
+# Generated by Django 3.1.4 on 2020-12-19 06:59
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('sync', '0007_auto_20201219_0645'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='source',
+            name='download_cap',
+            field=models.IntegerField(choices=[(0, 'No cap'), (604800, '1 week (7 days)'), (2592000, '1 month (30 days)'), (7776000, '3 months (90 days)'), (15552000, '6 months (180 days)'), (31536000, '1 year (365 days)'), (63072000, '2 years (730 days)'), (94608000, '3 years (1095 days)'), (157680000, '5 years (1825 days)'), (315360000, '10 years (3650 days)')], default=0, help_text='Do not download media older than this capped date', verbose_name='download cap'),
+        ),
+    ]

tubesync/sync/models.py

@@ -1,7 +1,9 @@
 import os
 import uuid
 import json
-from datetime import datetime
+from xml.etree import ElementTree
+from collections import OrderedDict
+from datetime import datetime, timedelta
 from pathlib import Path
 from django.conf import settings
 from django.db import models
@@ -123,6 +125,18 @@ class Source(models.Model):
         SOURCE_TYPE_YOUTUBE_PLAYLIST: 'id',
     }
 
+    class CapChoices(models.IntegerChoices):
+        CAP_NOCAP = 0, _('No cap')
+        CAP_7DAYS = 604800, _('1 week (7 days)')
+        CAP_30DAYS = 2592000, _('1 month (30 days)')
+        CAP_90DAYS = 7776000, _('3 months (90 days)')
+        CAP_6MONTHS = 15552000, _('6 months (180 days)')
+        CAP_1YEAR = 31536000, _('1 year (365 days)')
+        CAP_2YEARs = 63072000, _('2 years (730 days)')
+        CAP_3YEARs = 94608000, _('3 years (1095 days)')
+        CAP_5YEARs = 157680000, _('5 years (1825 days)')
+        CAP_10YEARS = 315360000, _('10 years (3650 days)')
+
     class IndexSchedule(models.IntegerChoices):
         EVERY_HOUR = 3600, _('Every hour')
         EVERY_2_HOURS = 7200, _('Every 2 hours')
@@ -195,6 +209,12 @@ class Source(models.Model):
         default=IndexSchedule.EVERY_6_HOURS,
         help_text=_('Schedule of how often to index the source for new media')
     )
+    download_cap = models.IntegerField(
+        _('download cap'),
+        choices=CapChoices.choices,
+        default=CapChoices.CAP_NOCAP,
+        help_text=_('Do not download media older than this capped date')
+    )
     delete_old_media = models.BooleanField(
         _('delete old media'),
         default=False,
@@ -253,6 +273,11 @@ class Source(models.Model):
         default=False,
         help_text=_('Copy thumbnails with the media, these may be detected and used by some media servers')
     )
+    write_nfo = models.BooleanField(
+        _('write nfo'),
+        default=False,
+        help_text=_('Write an NFO file in XML with the media info, these may be detected and used by some media servers')
+    )
     has_failed = models.BooleanField(
         _('has failed'),
         default=False,
@@ -283,6 +308,14 @@ class Source(models.Model):
     def is_video(self):
         return not self.is_audio
 
+    @property
+    def download_cap_date(self):
+        delta = self.download_cap
+        if delta > 0:
+            return timezone.now() - timedelta(seconds=delta)
+        else:
+            return False
+
     @property
     def extension(self):
         '''
@@ -370,12 +403,16 @@ class Source(models.Model):
             'yyyymmdd': timezone.now().strftime('%Y%m%d'),
             'yyyy_mm_dd': timezone.now().strftime('%Y-%m-%d'),
             'yyyy': timezone.now().strftime('%Y'),
+            'mm': timezone.now().strftime('%m'),
+            'dd': timezone.now().strftime('%d'),
             'source': self.slugname,
             'source_full': self.name,
             'title': 'some-media-title-name',
             'title_full': 'Some Media Title Name',
             'key': 'SoMeUnIqUiD',
             'format': '-'.join(fmt),
+            'playlist_index': 1,
+            'playlist_title': 'Some Playlist Title',
             'ext': self.extension,
             'resolution': self.source_resolution if self.source_resolution else '',
             'height': '720' if self.source_resolution else '',
@@ -389,7 +426,7 @@ class Source(models.Model):
     def get_example_media_format(self):
         try:
             return self.media_format.format(**self.example_media_format_dict)
-        except Exception:
+        except Exception as e:
             return ''
 
     def index_media(self):
@@ -474,7 +511,47 @@ class Media(models.Model):
             Source.SOURCE_TYPE_YOUTUBE_CHANNEL: 'formats',
             Source.SOURCE_TYPE_YOUTUBE_CHANNEL_ID: 'formats',
             Source.SOURCE_TYPE_YOUTUBE_PLAYLIST: 'formats',
-        }
+        },
+        'categories': {
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL: 'categories',
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL_ID: 'categories',
+            Source.SOURCE_TYPE_YOUTUBE_PLAYLIST: 'categories',
+        },
+        'rating': {
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL: 'average_rating',
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL_ID: 'average_rating',
+            Source.SOURCE_TYPE_YOUTUBE_PLAYLIST: 'average_rating',
+        },
+        'age_limit': {
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL: 'age_limit',
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL_ID: 'age_limit',
+            Source.SOURCE_TYPE_YOUTUBE_PLAYLIST: 'age_limit',
+        },
+        'uploader': {
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL: 'uploader',
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL_ID: 'uploader',
+            Source.SOURCE_TYPE_YOUTUBE_PLAYLIST: 'uploader',
+        },
+        'upvotes': {
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL: 'like_count',
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL_ID: 'like_count',
+            Source.SOURCE_TYPE_YOUTUBE_PLAYLIST: 'like_count',
+        },
+        'downvotes': {
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL: 'dislike_count',
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL_ID: 'dislike_count',
+            Source.SOURCE_TYPE_YOUTUBE_PLAYLIST: 'dislike_count',
+        },
+        'playlist_index': {
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL: 'playlist_index',
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL_ID: 'playlist_index',
+            Source.SOURCE_TYPE_YOUTUBE_PLAYLIST: 'playlist_index',
+        },
+        'playlist_title': {
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL: 'playlist_title',
+            Source.SOURCE_TYPE_YOUTUBE_CHANNEL_ID: 'playlist_title',
+            Source.SOURCE_TYPE_YOUTUBE_PLAYLIST: 'playlist_title',
+        },
     }
     STATE_UNKNOWN = 'unknown'
     STATE_SCHEDULED = 'scheduled'
@@ -805,12 +882,16 @@ class Media(models.Model):
             'yyyymmdd': dateobj.strftime('%Y%m%d'),
             'yyyy_mm_dd': dateobj.strftime('%Y-%m-%d'),
             'yyyy': dateobj.strftime('%Y'),
+            'mm': dateobj.strftime('%m'),
+            'dd': dateobj.strftime('%d'),
             'source': self.source.slugname,
             'source_full': self.source.name,
             'title': self.slugtitle,
             'title_full': self.title,
             'key': self.key,
             'format': '-'.join(display_format['format']),
+            'playlist_index': self.playlist_index,
+            'playlist_title': self.playlist_title,
             'ext': self.source.extension,
             'resolution': display_format['resolution'],
             'height': display_format['height'],
@@ -879,26 +960,78 @@ class Media(models.Model):
             return seconds_to_timestr(duration)
         return '??:??:??'
 
+    @property
+    def categories(self):
+        field = self.get_metadata_field('categories')
+        return self.loaded_metadata.get(field, [])
+
+    @property
+    def rating(self):
+        field = self.get_metadata_field('rating')
+        return self.loaded_metadata.get(field, 0)
+
+    @property
+    def votes(self):
+        field = self.get_metadata_field('upvotes')
+        upvotes = self.loaded_metadata.get(field, 0)
+        field = self.get_metadata_field('downvotes')
+        downvotes = self.loaded_metadata.get(field, 0)
+        return upvotes + downvotes
+
+    @property
+    def age_limit(self):
+        field = self.get_metadata_field('age_limit')
+        return self.loaded_metadata.get(field, 0)
+
+    @property
+    def uploader(self):
+        field = self.get_metadata_field('uploader')
+        return self.loaded_metadata.get(field, '')
+
     @property
     def formats(self):
         field = self.get_metadata_field('formats')
         return self.loaded_metadata.get(field, [])
 
+    @property
+    def playlist_index(self):
+        field = self.get_metadata_field('playlist_index')
+        return self.loaded_metadata.get(field, 0)
+
+    @property
+    def playlist_title(self):
+        field = self.get_metadata_field('playlist_title')
+        return self.loaded_metadata.get(field, '')
+
     @property
     def filename(self):
-        # If a media_file has been downloaded use its existing name
-        if self.media_file:
-            return os.path.basename(self.media_file.name)
         # Otherwise, create a suitable filename from the source media_format
         media_format = str(self.source.media_format)
         media_details = self.format_dict
         return media_format.format(**media_details)
 
+    @property
+    def thumbname(self):
+        filename = self.filename
+        prefix, ext = os.path.splitext(filename)
+        return f'{prefix}.jpg'
+
+    @property
+    def thumbpath(self):
+        return self.source.directory_path / self.thumbname
+
+    @property
+    def nfoname(self):
+        filename = self.filename
+        prefix, ext = os.path.splitext(filename)
+        return f'{prefix}.nfo'
+    
+    @property
+    def nfopath(self):
+        return self.source.directory_path / self.nfoname
+
     @property
     def directory_path(self):
-        # If a media_file has been downloaded use its existing directory
-        if self.media_file:
-            return os.path.dirname(self.media_file.name)
         # Otherwise, create a suitable filename from the source media_format
         media_format = str(self.source.media_format)
         media_details = self.format_dict
@@ -921,6 +1054,103 @@ class Media(models.Model):
             return False
         return os.path.exists(self.media_file.path)
 
+    @property
+    def nfoxml(self):
+        '''
+            Returns an NFO formatted (prettified) XML string.
+        '''
+        nfo = ElementTree.Element('episodedetails')
+        nfo.text = '\n  '
+        # title = media metadata title
+        title = nfo.makeelement('title', {})
+        title.text = str(self.name).strip()
+        title.tail = '\n  '
+        nfo.append(title)
+        # showtitle = source name
+        showtitle = nfo.makeelement('showtitle', {})
+        showtitle.text = str(self.source.name).strip()
+        showtitle.tail = '\n  '
+        nfo.append(showtitle)
+        # ratings = media metadata youtube rating
+        value = nfo.makeelement('value', {})
+        value.text = str(self.rating)
+        value.tail = '\n      '
+        votes = nfo.makeelement('votes', {})
+        votes.text = str(self.votes)
+        votes.tail = '\n    '
+        rating_attrs = OrderedDict()
+        rating_attrs['name'] = 'youtube'
+        rating_attrs['max'] = '5'
+        rating_attrs['default'] = 'True'
+        rating = nfo.makeelement('rating', rating_attrs)
+        rating.text = '\n      '
+        rating.append(value)
+        rating.append(votes)
+        rating.tail = '\n  '
+        ratings = nfo.makeelement('ratings', {})
+        ratings.text = '\n    '
+        ratings.append(rating)
+        ratings.tail = '\n  '
+        nfo.append(ratings)
+        # plot = media metadata description
+        plot = nfo.makeelement('plot', {})
+        plot.text = str(self.description).strip()
+        plot.tail = '\n  '
+        nfo.append(plot)
+        # thumb = local path to media thumbnail
+        thumb = nfo.makeelement('thumb', {})
+        thumb.text = self.thumbname if self.source.copy_thumbnails else ''
+        thumb.tail = '\n  '
+        nfo.append(thumb)
+        # mpaa = media metadata age requirement
+        mpaa = nfo.makeelement('mpaa', {})
+        mpaa.text = str(self.age_limit)
+        mpaa.tail = '\n  '
+        nfo.append(mpaa)
+        # runtime = media metadata duration in seconds
+        runtime = nfo.makeelement('runtime', {})
+        runtime.text = str(self.duration)
+        runtime.tail = '\n  '
+        nfo.append(runtime)
+        # id = media key
+        idn = nfo.makeelement('id', {})
+        idn.text = str(self.key).strip()
+        idn.tail = '\n  '
+        nfo.append(idn)
+        # uniqueid = media key
+        uniqueid_attrs = OrderedDict()
+        uniqueid_attrs['type'] = 'youtube'
+        uniqueid_attrs['default'] = 'True'
+        uniqueid = nfo.makeelement('uniqueid', uniqueid_attrs)
+        uniqueid.text = str(self.key).strip()
+        uniqueid.tail = '\n  '
+        nfo.append(uniqueid)
+        # studio = media metadata uploader
+        studio = nfo.makeelement('studio', {})
+        studio.text = str(self.uploader).strip()
+        studio.tail = '\n  '
+        nfo.append(studio)
+        # aired = media metadata uploaded date
+        aired = nfo.makeelement('aired', {})
+        upload_date = self.upload_date
+        aired.text = upload_date.strftime('%Y-%m-%d') if upload_date else ''
+        aired.tail = '\n  '
+        nfo.append(aired)
+        # dateadded = date and time media was created in tubesync
+        dateadded = nfo.makeelement('dateadded', {})
+        dateadded.text = self.created.strftime('%Y-%m-%d %H:%M:%S')
+        dateadded.tail = '\n  '
+        nfo.append(dateadded)
+        # genre = any media metadata categories if they exist
+        for category_str in self.categories:
+            genre = nfo.makeelement('genre', {})
+            genre.text = str(category_str).strip()
+            genre.tail = '\n  '
+            nfo.append(genre)
+        nfo[-1].tail = '\n'
+        # Return XML tree as a prettified string
+        return ElementTree.tostring(nfo, encoding='utf8', method='xml').decode('utf8')
+
     def get_download_state(self, task=None):
         if self.downloaded:
             return self.STATE_DOWNLOADED

tubesync/sync/signals.py

@@ -145,20 +145,6 @@ def media_pre_delete(sender, instance, **kwargs):
     if thumbnail_url:
         delete_task_by_media('sync.tasks.download_media_thumbnail',
                              (str(instance.pk), thumbnail_url))
-    # Delete media thumbnail if it exists
-    if instance.thumb:
-        log.info(f'Deleting thumbnail for: {instance} path: {instance.thumb.path}')
-        delete_file(instance.thumb.path)
-    # Delete the media file if it exists
-    if instance.media_file:
-        filepath = instance.media_file.path
-        log.info(f'Deleting media for: {instance} path: {filepath}')
-        delete_file(filepath)
-        # Delete thumbnail copy if it exists
-        barefilepath, fileext = os.path.splitext(filepath)
-        thumbpath = f'{barefilepath}.jpg'
-        log.info(f'Deleting thumbnail for: {instance} path: {thumbpath}')
-        delete_file(thumbpath)
 
 
 @receiver(post_delete, sender=Media)

tubesync/sync/tasks.py

@@ -23,7 +23,8 @@ from background_task.models import Task, CompletedTask
 from common.logger import log
 from common.errors import NoMediaException, DownloadFailedException
 from .models import Source, Media, MediaServer
-from .utils import get_remote_image, resize_image_to_height, delete_file
+from .utils import (get_remote_image, resize_image_to_height, delete_file,
+                    write_text_file)
 
 
 def get_hash(task_name, pk):
@@ -186,6 +187,14 @@ def index_source_task(source_id):
         else:
             log.error(f'Media has no upload date, skipping: {source} / {media}')
             continue
+        # If the source has a download cap date check the upload date is allowed
+        max_cap_age = source.download_cap_date
+        if max_cap_age:
+            if media.published < max_cap_age:
+                # Media was published after the cap date, skip it
+                log.warn(f'Media: {source} / {media} is older than cap age '
+                         f'{max_cap_age}, skipping')
+                continue
         # If the source has a cut-off check the upload date is within the allowed delta
         if source.delete_old_media and source.days_to_keep > 0:
             delta = timezone.now() - timedelta(days=source.days_to_keep)
@@ -317,11 +326,13 @@ def download_media(media_id):
         media.save()
         # If selected, copy the thumbnail over as well
         if media.source.copy_thumbnails and media.thumb:
-            barefilepath, fileext = os.path.splitext(filepath)
-            thumbpath = f'{barefilepath}.jpg'
             log.info(f'Copying media thumbnail from: {media.thumb.path} '
-                     f'to: {thumbpath}')
-            copyfile(media.thumb.path, thumbpath)
+                     f'to: {media.thumbpath}')
+            copyfile(media.thumb.path, media.thumbpath)
+        # If selected, write an NFO file
+        if media.source.write_nfo:
+            log.info(f'Writing media NFO file to: to: {media.nfopath}')
+            write_text_file(media.nfopath, media.nfoxml)
         # Schedule a task to update media servers
         for mediaserver in MediaServer.objects.all():
             log.info(f'Scheduling media server updates')

tubesync/sync/templates/sync/_mediaformatvars.html

@@ -11,18 +11,28 @@
     <tr>
       <td>{yyyymmdd}</td>
       <td>Media publish date in YYYYMMDD</td>
-      <td>20210101</td>
+      <td>20210131</td>
     </tr>
     <tr>
       <td>{yyyy_mm_dd}</td>
       <td>Media publish date in YYYY-MM-DD</td>
-      <td>2021-01-01</td>
+      <td>2021-01-31</td>
     </tr>
     <tr>
       <td>{yyyy}</td>
       <td>Media publish year in YYYY</td>
       <td>2021</td>
     </tr>
+    <tr>
+      <td>{mm}</td>
+      <td>Media publish year in MM</td>
+      <td>01</td>
+    </tr>
+    <tr>
+      <td>{dd}</td>
+      <td>Media publish year in DD</td>
+      <td>31</td>
+    </tr>
     <tr>
       <td>{source}</td>
       <td>Lower case source name, max 80 chars</td>
@@ -53,6 +63,16 @@
       <td>Media format string</td>
       <td>720p-avc1-mp4a</td>
     </tr>
+    <tr>
+      <td>{playlist_index}</td>
+      <td>Playlist index of media, if it's in a playlist</td>
+      <td>12</td>
+    </tr>
+    <tr>
+      <td>{playlist_title}</td>
+      <td>Playlist title of media, if it's in a playlist</td>
+      <td>Some Playlist</td>
+    </tr>
     <tr>
       <td>{ext}</td>
       <td>File extension</td>

tubesync/sync/templates/sync/dashboard.html

@@ -71,7 +71,7 @@
         <div class="collection">
         {% for media in latest_downloads %}
           <a href="{% url 'sync:media-item' pk=media.pk %}" class="collection-item">
-            <div class="truncate"><strong>{{ media.name }}</strong> ({{ media.source }})</div>
+            <div class="truncate"><strong>{{ media.name }}</strong></div>
             <div class="truncate"><strong>{{ media.download_date|timesince:now }}</strong> ago from &quot;{{ media.source.name }}&quot;</div>
           </a>
         {% empty %}
@@ -89,7 +89,7 @@
           {% for media in largest_downloads %}
             <a href="{% url 'sync:media-item' pk=media.pk %}" class="collection-item">
               <div class="truncate">{{ media.name }}</div>
-              <div class="truncate"><strong>{{ media.downloaded_filesize|filesizeformat }}</strong>{% if media.downloaded_format %} in {{ media.downloaded_format }}{% endif %}</div>
+              <div class="truncate"><strong>{{ media.downloaded_filesize|filesizeformat }}</strong>{% if media.downloaded_format %} in {{ media.downloaded_format }}{% endif %} from &quot;{{ media.source.name }}&quot;</div>
             </a>
           {% empty %}
             <span class="collection-item">No media has been downloaded.</span>

tubesync/sync/templates/sync/source-add.html

@@ -25,7 +25,7 @@
 </div>
 <div class="row">
   <div class="col s12">
-{% include 'mediaformatvars.html' %}
+{% include 'sync/_mediaformatvars.html' %}
   </div>
 </div>
 {% endblock %}

tubesync/sync/templates/sync/source-update.html

@@ -27,7 +27,7 @@
 </div>
 <div class="row">
   <div class="col s12">
-{% include 'mediaformatvars.html' %}
+{% include 'sync/_mediaformatvars.html' %}
   </div>
 </div>
 {% endblock %}

tubesync/sync/templates/sync/source.html

@@ -51,6 +51,12 @@
         <td class="hide-on-small-only">Example filename</td>
         <td><span class="hide-on-med-and-up">Example filename<br></span><strong>{{ source.get_example_media_format }}</strong></td>
       </tr>
+      {% if source.download_cap > 0 %}
+      <tr title="Do not download videos older than this cap">
+        <td class="hide-on-small-only">Download cap</td>
+        <td><span class="hide-on-med-and-up">Download cap<br></span><strong>{{ source.get_download_cap_display }}</strong></td>
+      </tr>
+      {% endif %}
       <tr title="Schedule of how often to index the source for new media">
         <td class="hide-on-small-only">Index schedule</td>
         <td><span class="hide-on-med-and-up">Index schedule<br></span><strong>{{ source.get_index_schedule_display }}</strong></td>
@@ -97,6 +103,10 @@
         <td class="hide-on-small-only">Copy thumbnails?</td>
         <td><span class="hide-on-med-and-up">Copy thumbnails?<br></span><strong>{% if source.copy_thumbnails %}<i class="fas fa-check"></i>{% else %}<i class="fas fa-times"></i>{% endif %}</strong></td>
       </tr>
+      <tr title="Should an NFO file be written with the media?">
+        <td class="hide-on-small-only">Write NFO?</td>
+        <td><span class="hide-on-med-and-up">Write NFO?<br></span><strong>{% if source.write_nfo %}<i class="fas fa-check"></i>{% else %}<i class="fas fa-times"></i>{% endif %}</strong></td>
+      </tr>
       {% if source.delete_old_media and source.days_to_keep > 0 %}
       <tr title="Days after which your media from this source will be locally deleted">
         <td class="hide-on-small-only">Delete old media</td>

tubesync/sync/testdata/metadata.json

@@ -3,15 +3,24 @@
     "upload_date":"20170911",
     "license":null,
     "creator":null,
-    "title":"no fancy stuff",
+    "title":"no fancy stuff title",
     "alt_title":null,
-    "description":"no fancy stuff",
-    "categories":[],
+    "description":"no fancy stuff desc",
+    "average_rating": 1.2345,
+    "dislike_count": 123,
+    "like_count": 456,
+    "playlist_index": 789,
+    "playlist_title": "test playlist",
+    "uploader": "test uploader",
+    "categories":[
+        "test category 1",
+        "test category 2"
+    ],
     "tags":[],
     "subtitles":{},
     "automatic_captions":{},
     "duration":401.0,
-    "age_limit":0,
+    "age_limit":50,
     "annotations":null,
     "chapters":null,
     "formats":[

tubesync/sync/tests.py

@@ -6,7 +6,9 @@
 
 
 import logging
+from datetime import datetime
 from urllib.parse import urlsplit
+from xml.etree import ElementTree
 from django.conf import settings
 from django.test import TestCase, Client
 from django.utils import timezone
@@ -161,6 +163,7 @@ class FrontEndTestCase(TestCase):
             'name': 'testname',
             'directory': 'testdirectory',
             'media_format': settings.MEDIA_FORMATSTR_DEFAULT,
+            'download_cap': 0,
             'index_schedule': 3600,
             'delete_old_media': False,
             'days_to_keep': 14,
@@ -201,6 +204,7 @@ class FrontEndTestCase(TestCase):
             'name': 'testname',
             'directory': 'testdirectory',
             'media_format': settings.MEDIA_FORMATSTR_DEFAULT,
+            'download_cap': 0,
             'index_schedule': Source.IndexSchedule.EVERY_HOUR,
             'delete_old_media': False,
             'days_to_keep': 14,
@@ -229,6 +233,7 @@ class FrontEndTestCase(TestCase):
             'name': 'testname',
             'directory': 'testdirectory',
             'media_format': settings.MEDIA_FORMATSTR_DEFAULT,
+            'download_cap': 0,
             'index_schedule': Source.IndexSchedule.EVERY_2_HOURS,  # changed
             'delete_old_media': False,
             'days_to_keep': 14,
@@ -430,7 +435,6 @@ class FrontEndTestCase(TestCase):
         response = c.get('/tasks-completed')
         self.assertEqual(response.status_code, 200)
 
-
     def test_mediasevrers(self):
         # Media servers overview page
         c = Client()
@@ -507,6 +511,12 @@ class FilepathTestCase(TestCase):
         self.source.media_format = 'test-{yyyy}'
         self.assertEqual(self.source.get_example_media_format(),
                          'test-' + timezone.now().strftime('%Y'))
+        self.source.media_format = 'test-{mm}'
+        self.assertEqual(self.source.get_example_media_format(),
+                         'test-' + timezone.now().strftime('%m'))
+        self.source.media_format = 'test-{dd}'
+        self.assertEqual(self.source.get_example_media_format(),
+                         'test-' + timezone.now().strftime('%d'))
         self.source.media_format = 'test-{source}'
         self.assertEqual(self.source.get_example_media_format(),
                          'test-' + self.source.slugname)
@@ -525,6 +535,12 @@ class FilepathTestCase(TestCase):
         self.source.media_format = 'test-{format}'
         self.assertEqual(self.source.get_example_media_format(),
                          'test-1080p-vp9-opus')
+        self.source.media_format = 'test-{playlist_index}'
+        self.assertEqual(self.source.get_example_media_format(),
+                         'test-1')
+        self.source.media_format = 'test-{playlist_title}'
+        self.assertEqual(self.source.get_example_media_format(),
+                         'test-Some Playlist Title')
         self.source.media_format = 'test-{ext}'
         self.assertEqual(self.source.get_example_media_format(),
                          'test-' + self.source.extension)
@@ -582,7 +598,79 @@ class FilepathTestCase(TestCase):
         self.source.media_format = ('{title}_{key}_{resolution}-{height}x{width}-'
                                     '{acodec}-{vcodec}-{fps}fps-{hdr}.{ext}')
         self.assertEqual(test_media.filename,
-                         'no-fancy-stuff_test_720p-720x1280-opus-vp9-30fps-hdr.mkv')
+                         ('no-fancy-stuff-title_test_720p-720x1280-opus'
+                          '-vp9-30fps-hdr.mkv'))
+
+
+class MediaTestCase(TestCase):
+
+    def setUp(self):
+        # Disable general logging for test case
+        logging.disable(logging.CRITICAL)
+        # Add a test source
+        self.source = Source.objects.create(
+            source_type=Source.SOURCE_TYPE_YOUTUBE_CHANNEL,
+            key='testkey',
+            name='testname',
+            directory='testdirectory',
+            media_format=settings.MEDIA_FORMATSTR_DEFAULT,
+            index_schedule=3600,
+            delete_old_media=False,
+            days_to_keep=14,
+            source_resolution=Source.SOURCE_RESOLUTION_1080P,
+            source_vcodec=Source.SOURCE_VCODEC_VP9,
+            source_acodec=Source.SOURCE_ACODEC_OPUS,
+            prefer_60fps=False,
+            prefer_hdr=False,
+            fallback=Source.FALLBACK_FAIL
+        )
+        # Add some test media
+        self.media = Media.objects.create(
+            key='mediakey',
+            source=self.source,
+            metadata=metadata,
+        )
+        # Fix a created datetime for predictable testing
+        self.media.created = datetime(year=2020, month=1, day=1, hour=1,
+                                      minute=1, second=1)
+
+    def test_nfo(self):
+        expected_nfo = [
+            "<?xml version='1.0' encoding='utf8'?>",
+            '<episodedetails>',
+            '  <title>no fancy stuff title</title>',
+            '  <showtitle>testname</showtitle>',
+            '  <ratings>',
+            '    <rating default="True" max="5" name="youtube">',
+            '      <value>1.2345</value>',
+            '      <votes>579</votes>',
+            '    </rating>',
+            '  </ratings>',
+            '  <plot>no fancy stuff desc</plot>',
+            '  <thumb />',  # media.thumbfile is empty without media existing
+            '  <mpaa>50</mpaa>',
+            '  <runtime>401</runtime>',
+            '  <id>mediakey</id>',
+            '  <uniqueid default="True" type="youtube">mediakey</uniqueid>',
+            '  <studio>test uploader</studio>',
+            '  <aired>2017-09-11</aired>',
+            '  <dateadded>2020-01-01 01:01:01</dateadded>',
+            '  <genre>test category 1</genre>',
+            '  <genre>test category 2</genre>',
+            '</episodedetails>',
+        ]
+        expected_tree = ElementTree.fromstring('\n'.join(expected_nfo))
+        nfo_tree = ElementTree.fromstring(self.media.nfoxml)
+        # Check each node with attribs in expected_tree is present in test_nfo
+        for expected_node in expected_tree:
+            # Ignore checking <genre>, only tag we may have multiple of
+            if expected_node.tag == 'genre':
+                continue
+            # Find the same node in the NFO XML tree
+            nfo_node = nfo_tree.find(expected_node.tag)
+            self.assertEqual(expected_node.attrib, nfo_node.attrib)
+            self.assertEqual(expected_node.tag, nfo_node.tag)
+            self.assertEqual(expected_node.text, nfo_node.text)
 
 
 class FormatMatchingTestCase(TestCase):

tubesync/sync/utils.py

@@ -108,6 +108,14 @@ def file_is_editable(filepath):
     return False
 
 
+def write_text_file(filepath, filedata):
+    if not isinstance(filedata, str):
+        raise ValueError(f'filedata must be a str, got "{type(filedata)}"')
+    with open(filepath, 'wt') as f:
+        bytes_written = f.write(filedata)
+    return bytes_written
+
+
 def delete_file(filepath):
     if file_is_editable(filepath):
         return os.remove(filepath)

tubesync/sync/views.py

@@ -274,9 +274,9 @@ class AddSourceView(CreateView):
     template_name = 'sync/source-add.html'
     model = Source
     fields = ('source_type', 'key', 'name', 'directory', 'media_format',
-              'index_schedule', 'delete_old_media', 'days_to_keep',
+              'index_schedule', 'download_cap', 'delete_old_media', 'days_to_keep',
               'source_resolution', 'source_vcodec', 'source_acodec', 'prefer_60fps',
-              'prefer_hdr', 'fallback', 'copy_thumbnails')
+              'prefer_hdr', 'fallback', 'copy_thumbnails', 'write_nfo')
     errors = {
         'invalid_media_format': _('Invalid media format, the media format contains '
                                   'errors or is empty. Check the table at the end of '
@@ -334,8 +334,8 @@ class SourceView(DetailView):
     messages = {
         'source-created': _('Your new source has been created. If you have added a '
                             'very large source such as a channel with hundreds of '
-                            'videos it can take several minutes for media to start '
-                            'to appear.'),
+                            'videos it can take several minutes or up to an hour '
+                            'for media to start to appear.'),
         'source-updated': _('Your source has been updated.'),
     }
 
@@ -365,9 +365,9 @@ class UpdateSourceView(UpdateView):
     template_name = 'sync/source-update.html'
     model = Source
     fields = ('source_type', 'key', 'name', 'directory', 'media_format',
-              'index_schedule', 'delete_old_media', 'days_to_keep',
+              'index_schedule', 'download_cap', 'delete_old_media', 'days_to_keep',
               'source_resolution', 'source_vcodec', 'source_acodec', 'prefer_60fps',
-              'prefer_hdr', 'fallback', 'copy_thumbnails')
+              'prefer_hdr', 'fallback', 'copy_thumbnails', 'write_nfo')
     errors = {
         'invalid_media_format': _('Invalid media format, the media format contains '
                                   'errors or is empty. Check the table at the end of '
@@ -411,7 +411,12 @@ class DeleteSourceView(DeleteView, FormMixin):
             source = self.get_object()
             for media in Media.objects.filter(source=source):
                 if media.media_file:
+                    # Delete the media file
                     delete_file(media.media_file.name)
+                    # Delete thumbnail copy if it exists
+                    delete_file(media.thumbpath)
+                    # Delete NFO file if it exists
+                    delete_file(media.nfopath)
         return super().post(request, *args, **kwargs)
 
     def get_success_url(self):
@@ -556,13 +561,12 @@ class MediaRedownloadView(FormView, SingleObjectMixin):
             self.object.thumb = None
         # If the media file exists on disk, delete it
         if self.object.media_file_exists:
-            filepath = self.object.media_file.path
-            delete_file(filepath)
+            delete_file(self.object.media_file.path)
             self.object.media_file = None
             # If the media has an associated thumbnail copied, also delete it
-            barefilepath, fileext = os.path.splitext(filepath)
-            thumbpath = f'{barefilepath}.jpg'
-            delete_file(thumbpath)
+            delete_file(self.object.thumbpath)
+            # If the media has an associated NFO file with it, also delete it
+            delete_file(self.object.nfopath)
         # Reset all download data
         self.object.downloaded = False
         self.object.downloaded_audio_codec = None
@@ -602,13 +606,12 @@ class MediaSkipView(FormView, SingleObjectMixin):
         delete_task_by_media('sync.tasks.download_media', (str(self.object.pk),))
         # If the media file exists on disk, delete it
         if self.object.media_file_exists:
-            filepath = self.object.media_file.path
             delete_file(self.object.media_file.path)
             self.object.media_file = None
             # If the media has an associated thumbnail copied, also delete it
-            barefilepath, fileext = os.path.splitext(filepath)
-            thumbpath = f'{barefilepath}.jpg'
-            delete_file(thumbpath)
+            delete_file(self.object.thumbpath)
+            # If the media has an associated NFO file with it, also delete it
+            delete_file(self.object.nfopath)
         # Reset all download data
         self.object.downloaded = False
         self.object.downloaded_audio_codec = None

tubesync/tubesync/local_settings.py.container

@@ -15,6 +15,7 @@ SECRET_KEY = str(os.getenv('DJANGO_SECRET_KEY', 'tubesync-django-secret'))
 ALLOWED_HOSTS_STR = str(os.getenv('TUBESYNC_HOSTS', '127.0.0.1,localhost'))
 ALLOWED_HOSTS = ALLOWED_HOSTS_STR.split(',')
 DEBUG = True if os.getenv('TUBESYNC_DEBUG', False) else False
+FORCE_SCRIPT_NAME = os.getenv('DJANGO_FORCE_SCRIPT_NAME', None)
 
 
 TIME_ZONE = os.getenv('TZ', 'UTC')

tubesync/tubesync/settings.py

@@ -6,7 +6,7 @@ CONFIG_BASE_DIR = BASE_DIR
 DOWNLOADS_BASE_DIR = BASE_DIR
 
 
-VERSION = 0.5
+VERSION = 0.6
 SECRET_KEY = ''
 DEBUG = False
 ALLOWED_HOSTS = []
@@ -41,6 +41,7 @@ MIDDLEWARE = [
 
 
 ROOT_URLCONF = 'tubesync.urls'
+FORCE_SCRIPT_NAME = None
 
 
 TEMPLATES = [