Update changelog, skip docker for now until it is ready

CHANGELOG.md

@@ -4,16 +4,19 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
-## [v0.6.0] - 2023-03-09
+## [v0.6.0] - 2023-03-15
 
 - Configurable destinations for downloads
+  - Multiple destination directories can be configured
+  - When queueing a download, an alternate destination can be selected
 - When downloading from a playlist, show the total number of videos and how many have been downloaded
 - Show version in web UI
 - Improve index page (show URL of queued downloads instead of nothing)
-- Add docker support
 - Fixes and improvements to capturing output info and showing it in the UI
+- Show all log output in the popup
 - Fixes to handling of queued downloads
 - Fix portable mode to look in binary directory, not current directory
+- Automatically cleanup download list, removing old entries automatically
 
 ## [v0.5.5] - 2022-04-09
 

README.md

@@ -16,9 +16,10 @@ A frontend to youtube-dl (or compatible forks, like yt-dlp) to download videos w
 
 ## Binaries
 
-Binaries are available at https://github.com/tardisx/gropple/releases
+Binaries are available at <https://github.com/tardisx/gropple/releases>
 
-Gropple will automatically check for available updates and prompt you to upgrade.
+Gropple will automatically check for available updates and prompt you to
+upgrade.
 
 ## Running
 
@@ -32,66 +33,75 @@ interface. The address will be printed after startup:
 
 ## Using
 
-Bring up `http://localhost:6283` (or your configured address) in your browser. You 
+Bring up `http://localhost:6283` (or your configured address) in your browser.
-should see a link to the bookmarklet at the top of the screen, and the list of
+You should see a link to the bookmarklet at the top of the screen, and the list
-downloads (currently empty).
+of downloads (currently empty).
 
-Drag the bookmarklet to your favourites bar, or otherwise bookmark it as you 
+Drag the bookmarklet to your favourites bar, or otherwise bookmark it as you see
-see fit. Any kind of browser bookmark should work. The bookmarklet contains
+fit. Any kind of browser bookmark should work. The bookmarklet contains embedded
-embedded javascript to pass the URL of whatever page you are currently on back
+javascript to pass the URL of whatever page you are currently on back to
-to gropple.
+gropple.
 
-So, whenever you are on a page with a video you would like to download just 
+Whenever you are on a page with a video you would like to download just click
-click the bookmarklet.
+the bookmarklet.
 
-A popup window will appear. Choose a download profile and the download will start.
+A popup window will appear. Choose a download profile and the download will
-The status will be shown in the window, updating in real time.
+start. The status will be shown in the window, updating in real time.
 
 You may close this window at any time without stopping the download, the status
 of all downloads is available on the index page.
 
 ## Configuration
 
-Click the "config" link on the index page to configure gropple. The default options 
+Click the "config" link on the index page to configure gropple. The default
-are fine if you are running on your local machine. If you are running it remotely
+options are fine if you are running on your local machine. If you are running it
-you will need to set the "server address" to ensure the bookmarklet has the correct
+remotely you will need to set the "server address" to ensure the bookmarklet has
-URL in it.
+the correct URL in it.
 
 ### Configuring Downloaders
 
-Gropple's default configuration uses the original youtube-dl and has two profiles set
+Gropple's default configuration uses `yt-dlp` and has two profiles set up, one
-up, one for downloading video, the other for downloading audio (mp3).
+for downloading video, the other for downloading audio (mp3).
 
-Note that gropple does not include any downloaders, you have to install them separately.
+Note that gropple does not include any downloaders, you have to install them
+separately.
 
-If you would like to use a youtube-dl fork (like [yt-dlp](https://github.com/yt-dlp/yt-dlp))
+If you would like to use a youtube-dl compatible fork or change the options you
-or change the options, you can do so on the right hand side. Create as many profiles as you 
+can do so on the right hand side. Create as many profiles as you wish, whenever
-wish, whenever you start a download you can choose the appropriate profile.
+you start a download you can choose the appropriate profile.
 
-Note that the command arguments must each be specified separately - see the default configuration
+Note that the command arguments must each be specified separately - see the
-for an example.
+default configuration for an example.
 
-While gropple will use your `PATH` to find the executable, you can also specify a full path
+While gropple will use your `PATH` to find the executable, you can also specify
-instead. Note that any tools that the downloader calls itself (for instance, ffmpeg) will 
+a full path instead. Note that any tools that the downloader calls itself (for
-probably need to be available on your path.
+instance, `ffmpeg`) will need to be available on your path.
+
+### Alternate destinations
+
+Gropple supports adding additional optional destinations. By default, all
+downloads will be stored in the main download path specified in the config. You
+can also add one or more destinations, and you can choose one of these
+destinations when queueing a new download, or while it is still downloading from
+the popup.
+
+The file will be moved after downloading is complete.
 
 ## Portable mode
 
-If you'd like to use gropple from a USB stick or similar, copy the config file from
+If you'd like to use gropple from a USB stick or similar, copy the config file
-it's default location (shown when you start gropple) to the same location as the binary, and rename it to `gropple.yml`. 
+from its default location (shown when you start gropple) to the same location as
-
+the binary, and rename it to `gropple.yml`.
-If that file is present in the same directory as the binary, it will be used instead.
 
 ## Problems
 
-Most download problems are probably diagnosable via the log - check in the popup window and scroll
+Many download problems are diagnosable via the log - check in the popup window
-the log down to the bottom. The most common problem is that youtube-dl cannot be found, or its
+and scroll the log down to the bottom. The most common problem is that `yt-dlp`
-dependency (like ffmpeg) cannot be found on your path. 
+cannot be found, or its dependency (like `ffmpeg`) cannot be found on your path.
 
 For other problems, please file an issue on github.
 
 ## TODO
 
-Many things. Please raise an issue after checking the [currently open issues](https://github.com/tardisx/gropple/issues).
+Many things. Please raise an issue after checking the [currently open
-
+issues](https://github.com/tardisx/gropple/issues).
-

build_release.pl

@@ -40,6 +40,7 @@ foreach my $type (keys %build) {
 }
 
 # now docker
+exit 0;
 $ENV{VERSION}="$version";
 system "docker-compose", "-f", "docker-compose.build.yml", "build";
 system "docker", "tag", "tardisx/gropple:$version", "tardisx/gropple:latest";

config/config.go

@@ -57,6 +57,7 @@ type ConfigService struct {
 
 func (cs *ConfigService) LoadTestConfig() {
 	cs.LoadDefaultConfig()
+	cs.Config.Server.DownloadPath = "/tmp"
 	cs.Config.DownloadProfiles = []DownloadProfile{{Name: "test profile", Command: "sleep", Args: []string{"5"}}}
 }
 
@@ -93,7 +94,6 @@ func (cs *ConfigService) LoadDefaultConfig() {
 
 	cs.Config = &defaultConfig
 
-	return
 }
 
 func (c *Config) ProfileCalled(name string) *DownloadProfile {
@@ -318,6 +318,9 @@ func (cs *ConfigService) WriteConfig() {
 	}
 	defer file.Close()
 
-	file.Write(s)
+	_, err = file.Write(s)
+	if err != nil {
+		log.Fatalf("could not write config file %s: %s", path, err)
+	}
 	file.Close()
 }

download/download.go

@@ -1,12 +1,14 @@
 package download
 
 import (
+	"encoding/json"
 	"fmt"
 	"io"
 	"log"
 	"net/url"
 	"os"
 	"os/exec"
+	"path/filepath"
 	"regexp"
 	"strconv"
 	"strings"
@@ -23,7 +25,7 @@ type Download struct {
 	PopupUrl        string                 `json:"popup_url"`
 	Process         *os.Process            `json:"-"`
 	ExitCode        int                    `json:"exit_code"`
-	State           string                 `json:"state"`
+	State           State                  `json:"state"`
 	DownloadProfile config.DownloadProfile `json:"download_profile"`
 	Destination     *config.Destination    `json:"destination"`
 	Finished        bool                   `json:"finished"`
@@ -45,6 +47,32 @@ type Manager struct {
 	Lock         sync.Mutex
 }
 
+func (m *Manager) String() string {
+	m.Lock.Lock()
+	defer m.Lock.Unlock()
+	out := fmt.Sprintf("Max per domain: %d, downloads: %d\n", m.MaxPerDomain, len(m.Downloads))
+
+	for _, dl := range m.Downloads {
+		out = out + fmt.Sprintf("%3d: (%10s) %30s\n", dl.Id, dl.State, dl.Url)
+	}
+
+	return out
+
+}
+
+type State string
+
+const (
+	STATE_PREPARING            State = "Preparing to start"
+	STATE_CHOOSE_PROFILE       State = "Choose Profile"
+	STATE_QUEUED               State = "Queued"
+	STATE_DOWNLOADING          State = "Downloading"
+	STATE_DOWNLOADING_METADATA State = "Downloading metadata"
+	STATE_FAILED               State = "Failed"
+	STATE_COMPLETE             State = "Complete"
+	STATE_MOVED                State = "Moved"
+)
+
 var CanStopDownload = false
 
 var downloadId int32 = 0
@@ -54,13 +82,52 @@ func (m *Manager) ManageQueue() {
 		m.Lock.Lock()
 
 		m.startQueued(m.MaxPerDomain)
-		// m.cleanup()
+		m.moveToDest()
+		m.cleanup()
 		m.Lock.Unlock()
 
 		time.Sleep(time.Second)
 	}
 }
 
+func (m *Manager) DownloadsAsJSON() ([]byte, error) {
+
+	m.Lock.Lock()
+	defer m.Lock.Unlock()
+	for _, dl := range m.Downloads {
+		dl.Lock.Lock()
+		defer dl.Lock.Unlock()
+	}
+	b, err := json.Marshal(m.Downloads)
+	return b, err
+}
+
+func (m *Manager) moveToDest() {
+
+	// move any downloads that are complete and have a dest
+	for _, dl := range m.Downloads {
+
+		dl.Lock.Lock()
+		if dl.Destination != nil && dl.State == STATE_COMPLETE {
+			dl.State = STATE_MOVED
+			for _, fn := range dl.Files {
+				src := filepath.Join(dl.Config.Server.DownloadPath, fn)
+				dst := filepath.Join(dl.Destination.Path, fn)
+				err := os.Rename(src, dst)
+				if err != nil {
+					log.Printf("%s", err)
+					dl.Log = append(dl.Log, fmt.Sprintf("Could not move %s to %s - %s", fn, dl.Destination.Path, err))
+					break
+				} else {
+					dl.Log = append(dl.Log, fmt.Sprintf("Moved %s to %s", fn, dl.Destination.Path))
+
+				}
+			}
+		}
+		dl.Lock.Unlock()
+	}
+}
+
 // startQueued starts any downloads that have been queued, we would not exceed
 // maxRunning. If maxRunning is 0, there is no limit.
 func (m *Manager) startQueued(maxRunning int) {
@@ -70,7 +137,7 @@ func (m *Manager) startQueued(maxRunning int) {
 	for _, dl := range m.Downloads {
 		dl.Lock.Lock()
 
-		if dl.State == "downloading" || dl.State == "preparing to start" {
+		if dl.State == STATE_DOWNLOADING || dl.State == STATE_PREPARING {
 			active[dl.domain()]++
 		}
 		dl.Lock.Unlock()
@@ -81,8 +148,8 @@ func (m *Manager) startQueued(maxRunning int) {
 
 		dl.Lock.Lock()
 
-		if dl.State == "queued" && (maxRunning == 0 || active[dl.domain()] < maxRunning) {
+		if dl.State == STATE_QUEUED && (maxRunning == 0 || active[dl.domain()] < maxRunning) {
-			dl.State = "preparing to start"
+			dl.State = STATE_PREPARING
 			active[dl.domain()]++
 			log.Printf("Starting download for id:%d (%s)", dl.Id, dl.Url)
 
@@ -100,16 +167,17 @@ func (m *Manager) startQueued(maxRunning int) {
 }
 
 // cleanup removes old downloads from the list. Hardcoded to remove them one hour
-// completion.
+// completion. Expects the Manager to be locked.
-func (m *Manager) XXXcleanup() {
+func (m *Manager) cleanup() {
 	newDLs := []*Download{}
 	for _, dl := range m.Downloads {
-
+		dl.Lock.Lock()
 		if dl.Finished && time.Since(dl.FinishedTS) > time.Duration(time.Hour) {
 			// do nothing
 		} else {
 			newDLs = append(newDLs, dl)
 		}
+		dl.Lock.Unlock()
 
 	}
 	m.Downloads = newDLs
@@ -131,7 +199,18 @@ func (m *Manager) GetDlById(id int) (*Download, error) {
 func (m *Manager) Queue(dl *Download) {
 	dl.Lock.Lock()
 	defer dl.Lock.Unlock()
-	dl.State = "queued"
+	dl.State = STATE_QUEUED
+}
+
+func (m *Manager) ChangeDestination(dl *Download, dest *config.Destination) {
+	dl.Lock.Lock()
+	// we can only change destination is certain cases...
+	if dl.State != STATE_FAILED && dl.State != STATE_MOVED {
+		dl.Destination = dest
+	}
+
+	dl.Lock.Unlock()
+
 }
 
 func NewDownload(url string, conf *config.Config) *Download {
@@ -140,7 +219,7 @@ func NewDownload(url string, conf *config.Config) *Download {
 		Id:       int(downloadId),
 		Url:      url,
 		PopupUrl: fmt.Sprintf("/fetch/%d", int(downloadId)),
-		State:    "choose profile",
+		State:    STATE_CHOOSE_PROFILE,
 		Files:    []string{},
 		Log:      []string{},
 		Config:   conf,
@@ -153,7 +232,6 @@ func (m *Manager) AddDownload(dl *Download) {
 	m.Lock.Lock()
 	defer m.Lock.Unlock()
 	m.Downloads = append(m.Downloads, dl)
-	return
 }
 
 // func (dl *Download) AppendLog(text string) {
@@ -173,7 +251,10 @@ func (dl *Download) Stop() {
 	dl.Lock.Lock()
 	defer dl.Lock.Unlock()
 	dl.Log = append(dl.Log, "aborted by user")
-	dl.Process.Kill()
+	err := dl.Process.Kill()
+	if err != nil {
+		log.Printf("could not send kill to process: %s", err)
+	}
 }
 
 // domain returns a domain for this Download. Download should be locked.
@@ -194,7 +275,7 @@ func (dl *Download) domain() string {
 func (dl *Download) Begin() {
 	dl.Lock.Lock()
 
-	dl.State = "downloading"
+	dl.State = STATE_DOWNLOADING
 	cmdSlice := []string{}
 	cmdSlice = append(cmdSlice, dl.DownloadProfile.Args...)
 
@@ -208,7 +289,7 @@ func (dl *Download) Begin() {
 
 	stdout, err := cmd.StdoutPipe()
 	if err != nil {
-		dl.State = "failed"
+		dl.State = STATE_FAILED
 		dl.Finished = true
 		dl.FinishedTS = time.Now()
 		dl.Log = append(dl.Log, fmt.Sprintf("error setting up stdout pipe: %v", err))
@@ -219,7 +300,7 @@ func (dl *Download) Begin() {
 
 	stderr, err := cmd.StderrPipe()
 	if err != nil {
-		dl.State = "failed"
+		dl.State = STATE_FAILED
 		dl.Finished = true
 		dl.FinishedTS = time.Now()
 		dl.Log = append(dl.Log, fmt.Sprintf("error setting up stderr pipe: %v", err))
@@ -231,7 +312,7 @@ func (dl *Download) Begin() {
 	log.Printf("Executing command: %v", cmd)
 	err = cmd.Start()
 	if err != nil {
-		dl.State = "failed"
+		dl.State = STATE_FAILED
 		dl.Finished = true
 		dl.FinishedTS = time.Now()
 		dl.Log = append(dl.Log, fmt.Sprintf("error starting command '%s': %v", dl.DownloadProfile.Command, err))
@@ -258,19 +339,30 @@ func (dl *Download) Begin() {
 	}()
 
 	wg.Wait()
-	cmd.Wait()
 
+	err = cmd.Wait()
 	dl.Lock.Lock()
 
-	log.Printf("Process finished for id: %d (%v)", dl.Id, cmd)
+	if err != nil {
+		log.Printf("process failed for id: %d: %s", dl.Id, err)
 
-	dl.State = "complete"
+		dl.State = STATE_FAILED
+		dl.Finished = true
+		dl.FinishedTS = time.Now()
+		dl.ExitCode = cmd.ProcessState.ExitCode()
+
+	} else {
+
+		log.Printf("process finished for id: %d (%v)", dl.Id, cmd)
+
+		dl.State = STATE_COMPLETE
 		dl.Finished = true
 		dl.FinishedTS = time.Now()
 		dl.ExitCode = cmd.ProcessState.ExitCode()
 
 		if dl.ExitCode != 0 {
-		dl.State = "failed"
+			dl.State = STATE_FAILED
+		}
 	}
 	dl.Lock.Unlock()
 
@@ -318,7 +410,7 @@ func (dl *Download) updateMetadata(s string) {
 	matches := etaRE.FindStringSubmatch(s)
 	if len(matches) == 2 {
 		dl.Eta = matches[1]
-		dl.State = "downloading"
+		dl.State = STATE_DOWNLOADING
 
 	}
 
@@ -379,7 +471,7 @@ func (dl *Download) updateMetadata(s string) {
 	metadataDL := regexp.MustCompile(`Downloading JSON metadata page (\d+)`)
 	matches = metadataDL.FindStringSubmatch(s)
 	if len(matches) == 2 {
-		dl.State = "Downloading metadata, page " + matches[1]
+		dl.State = STATE_DOWNLOADING_METADATA
 	}
 
 	// [FixupM3u8] Fixing MPEG-TS in MP4 container of "file [-168849776_456239489].mp4"

download/download_notestdata.go

@@ -0,0 +1,8 @@
+//go:build !testdata
+
+package download
+
+import "github.com/tardisx/gropple/config"
+
+func (m *Manager) AddStressTestData(c *config.ConfigService) {
+}

download/download_test.go

@@ -2,7 +2,11 @@ package download
 
 import (
 	"strings"
+	"sync"
 	"testing"
+	"time"
+
+	"github.com/tardisx/gropple/config"
 )
 
 func TestUpdateMetadata(t *testing.T) {
@@ -76,79 +80,158 @@ func TestUpdateMetadata(t *testing.T) {
 // [download] 100% of 4.64MiB in 00:00
 // [ffmpeg] Merging formats into "Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.mp4"
 
-// func TestQueue(t *testing.T) {
+func TestQueue(t *testing.T) {
-// 	cs := config.ConfigService{}
+	cs := config.ConfigService{}
-// 	cs.LoadTestConfig()
+	cs.LoadTestConfig()
-// 	conf := cs.Config
+	conf := cs.Config
 
-// 	new1 := Download{Id: 1, Url: "http://sub.example.org/foo1", State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
+	new1 := NewDownload("http://sub.example.org/foo1", conf)
-// 	new2 := Download{Id: 2, Url: "http://sub.example.org/foo2", State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
+	new2 := NewDownload("http://sub.example.org/foo2", conf)
-// 	new3 := Download{Id: 3, Url: "http://sub.example.org/foo3", State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
+	new3 := NewDownload("http://sub.example.org/foo3", conf)
-// 	new4 := Download{Id: 4, Url: "http://example.org/", State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
+	new4 := NewDownload("http://example.org/", conf)
 
-// 	dls := Downloads{&new1, &new2, &new3, &new4}
+	// pretend the user chose a profile for each
-// 	dls.StartQueued(1)
+	new1.DownloadProfile = *conf.ProfileCalled("test profile")
-// 	time.Sleep(time.Millisecond * 100)
+	new2.DownloadProfile = *conf.ProfileCalled("test profile")
-// 	if dls[0].State == "queued" {
+	new3.DownloadProfile = *conf.ProfileCalled("test profile")
-// 		t.Error("#1 was not started")
+	new4.DownloadProfile = *conf.ProfileCalled("test profile")
-// 	}
+	new1.State = STATE_QUEUED
-// 	if dls[1].State != "queued" {
+	new2.State = STATE_QUEUED
-// 		t.Error("#2 is not queued")
+	new3.State = STATE_QUEUED
-// 	}
+	new4.State = STATE_QUEUED
-// 	if dls[3].State == "queued" {
-// 		t.Error("#4 is not started")
-// 	}
 
-// 	// this should start no more, as one is still going
+	q := Manager{
-// 	dls.StartQueued(1)
+		Downloads:    []*Download{},
-// 	time.Sleep(time.Millisecond * 100)
+		MaxPerDomain: 2,
-// 	if dls[1].State != "queued" {
+		Lock:         sync.Mutex{},
-// 		t.Error("#2 was started when it should not be")
+	}
-// 	}
 
-// 	dls.StartQueued(2)
+	q.AddDownload(new1)
-// 	time.Sleep(time.Millisecond * 100)
+	q.AddDownload(new2)
-// 	if dls[1].State == "queued" {
+	q.AddDownload(new3)
-// 		t.Error("#2 was not started but it should be")
+	q.AddDownload(new4)
 
-// 	}
+	q.startQueued(1)
 
-// 	dls.StartQueued(2)
+	// two should start, one from each of the two domains
-// 	time.Sleep(time.Millisecond * 100)
+	time.Sleep(time.Millisecond * 100)
-// 	if dls[3].State == "queued" {
+	if q.Downloads[0].State != STATE_DOWNLOADING {
-// 		t.Error("#4 was not started but it should be")
+		t.Errorf("#1 was not downloading - %s instead ", q.Downloads[0].State)
-// 	}
+		t.Log(q.String())
+	}
+	if q.Downloads[1].State != STATE_QUEUED {
+		t.Errorf("#2 is not queued - %s instead", q.Downloads[1].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[2].State != STATE_QUEUED {
+		t.Errorf("#3 is not queued - %s instead", q.Downloads[2].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[3].State != STATE_DOWNLOADING {
+		t.Errorf("#4 is not downloading - %s instead", q.Downloads[3].State)
+		t.Log(q.String())
+	}
 
-// 	// reset them all
+	// this should start no more, as one is still going
-// 	dls[0].State = "queued"
+	q.startQueued(1)
-// 	dls[1].State = "queued"
+	time.Sleep(time.Millisecond * 100)
-// 	dls[2].State = "queued"
+	if q.Downloads[0].State != STATE_DOWNLOADING {
-// 	dls[3].State = "queued"
+		t.Errorf("#1 was not downloading - %s instead ", q.Downloads[0].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[1].State != STATE_QUEUED {
+		t.Errorf("#2 is not queued - %s instead", q.Downloads[1].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[2].State != STATE_QUEUED {
+		t.Errorf("#3 is not queued - %s instead", q.Downloads[2].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[3].State != STATE_DOWNLOADING {
+		t.Errorf("#4 is not downloading - %s instead", q.Downloads[3].State)
+		t.Log(q.String())
+	}
 
-// 	dls.StartQueued(0)
+	// wait until the two finish, check
-// 	time.Sleep(time.Millisecond * 100)
+	time.Sleep(time.Second * 5.0)
+	if q.Downloads[0].State != STATE_COMPLETE {
+		t.Errorf("#1 was not complete - %s instead ", q.Downloads[0].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[1].State != STATE_QUEUED {
+		t.Errorf("#2 is not queued - %s instead", q.Downloads[1].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[2].State != STATE_QUEUED {
+		t.Errorf("#3 is not queued - %s instead", q.Downloads[2].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[3].State != STATE_COMPLETE {
+		t.Errorf("#4 is not complete - %s instead", q.Downloads[3].State)
+		t.Log(q.String())
+	}
 
-// 	// they should all be going
+	// this should start one more, as one is still going
-// 	if dls[0].State == "queued" || dls[1].State == "queued" || dls[2].State == "queued" || dls[3].State == "queued" {
+	q.startQueued(1)
-// 		t.Error("none should be queued")
+	time.Sleep(time.Millisecond * 100)
-// 	}
+	if q.Downloads[0].State != STATE_COMPLETE {
+		t.Errorf("#1 was not complete - %s instead ", q.Downloads[0].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[1].State != STATE_DOWNLOADING {
+		t.Errorf("#2 is not downloading - %s instead", q.Downloads[1].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[2].State != STATE_QUEUED {
+		t.Errorf("#3 is not queued - %s instead", q.Downloads[2].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[3].State != STATE_COMPLETE {
+		t.Errorf("#4 is not complete - %s instead", q.Downloads[3].State)
+		t.Log(q.String())
+	}
 
-// 	// reset them all
+	// this should start no more, as one is still going
-// 	dls[0].State = "queued"
+	q.startQueued(1)
-// 	dls[1].State = "queued"
+	time.Sleep(time.Millisecond * 100)
-// 	dls[2].State = "queued"
+	if q.Downloads[0].State != STATE_COMPLETE {
-// 	dls[3].State = "queued"
+		t.Errorf("#1 was not complete - %s instead ", q.Downloads[0].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[1].State != STATE_DOWNLOADING {
+		t.Errorf("#2 is not downloading - %s instead", q.Downloads[1].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[2].State != STATE_QUEUED {
+		t.Errorf("#3 is not queued - %s instead", q.Downloads[2].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[3].State != STATE_COMPLETE {
+		t.Errorf("#4 is not complete - %s instead", q.Downloads[3].State)
+		t.Log(q.String())
+	}
 
-// 	dls.StartQueued(2)
+	// but if we allow two per domain, the other queued one will start
-// 	time.Sleep(time.Millisecond * 100)
+	q.startQueued(2)
+	time.Sleep(time.Millisecond * 100)
+	if q.Downloads[0].State != STATE_COMPLETE {
+		t.Errorf("#1 was not complete - %s instead ", q.Downloads[0].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[1].State != STATE_DOWNLOADING {
+		t.Errorf("#2 is not downloading - %s instead", q.Downloads[1].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[2].State != STATE_DOWNLOADING {
+		t.Errorf("#3 is not downloading - %s instead", q.Downloads[2].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[3].State != STATE_COMPLETE {
+		t.Errorf("#4 is not complete - %s instead", q.Downloads[3].State)
+		t.Log(q.String())
+	}
 
-// 	// first two should be running, third not (same domain) and 4th running (different domain)
+}
-// 	if dls[0].State == "queued" || dls[1].State == "queued" || dls[2].State != "queued" || dls[3].State == "queued" {
-// 		t.Error("incorrect queued")
-
-// 	}
-
-// }
 
 func TestUpdateMetadataPlaylist(t *testing.T) {
 

download/download_testdata.go

@@ -0,0 +1,28 @@
+//go:build testdata
+
+package download
+
+import "github.com/tardisx/gropple/config"
+
+func (m *Manager) AddStressTestData(c *config.ConfigService) {
+
+	urls := []string{
+		"https://www.youtube.com/watch?v=qG_rRkuGBW8",
+		"https://www.youtube.com/watch?v=ZUzhZpQAU40",
+		"https://www.youtube.com/watch?v=kVxM3eRWGak",
+		"https://www.youtube.com/watch?v=pl-y9869y0w",
+		"https://vimeo.com/783453809",
+		"https://www.youtube.com/watch?v=Uw4NEPE4l3A",
+		"https://www.youtube.com/watch?v=2RF0lcTuuYE",
+		"https://www.youtube.com/watch?v=lymwNQY0dus",
+		"https://www.youtube.com/watch?v=NTc-I4Z_duc",
+		"https://www.youtube.com/watch?v=wNSm1TJ84Ac",
+		"https://vimeo.com/786570322",
+	}
+	for _, u := range urls {
+		d := NewDownload(u, c.Config)
+		d.DownloadProfile = *c.Config.ProfileCalled("standard video")
+		m.AddDownload(d)
+		m.Queue(d)
+	}
+}

go.mod

@@ -4,6 +4,6 @@ go 1.20
 
 require (
 	github.com/gorilla/mux v1.8.0
-	golang.org/x/mod v0.5.1
+	golang.org/x/mod v0.9.0
 	gopkg.in/yaml.v2 v2.4.0
 )

go.sum

@@ -1,7 +1,7 @@
 github.com/gorilla/mux v1.8.0 h1:i40aqfkR1h2SlN9hojwV5ZA91wcXFOvkdNIeFDP5koI=
 github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So=
-golang.org/x/mod v0.5.1 h1:OJxoQ/rynoF0dcCdI7cLPktw/hR2cueqYfjm43oqK38=
+golang.org/x/mod v0.9.0 h1:KENHtAZL2y3NLMYZeHY9DW8HW8V+kQyJsY/V9JlKvCs=
-golang.org/x/mod v0.5.1/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro=
+golang.org/x/mod v0.9.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=

main.go

@@ -21,11 +21,10 @@ import (
 )
 
 var dm *download.Manager
-var downloadId = 0
 var configService *config.ConfigService
 
 var versionInfo = version.Manager{
-	VersionInfo: version.Info{CurrentVersion: "v0.6.0-alpha.1"},
+	VersionInfo: version.Info{CurrentVersion: "v0.6.0-alpha.4"},
 }
 
 //go:embed web
@@ -102,7 +101,10 @@ func main() {
 	// check for a new version every 4 hours
 	go func() {
 		for {
-			versionInfo.UpdateGitHubVersion()
+			err := versionInfo.UpdateGitHubVersion()
+			if err != nil {
+				log.Printf("could not get version info: %s", err)
+			}
 			time.Sleep(time.Hour * 4)
 		}
 	}()
@@ -110,25 +112,7 @@ func main() {
 	// start downloading queued downloads when slots available, and clean up
 	// old entries
 	go dm.ManageQueue()
-
+	dm.AddStressTestData(configService)
-	urls := []string{
-		"https://www.youtube.com/watch?v=qG_rRkuGBW8",
-		"https://www.youtube.com/watch?v=ZUzhZpQAU40",
-		// "https://www.youtube.com/watch?v=kVxM3eRWGak",
-		// "https://www.youtube.com/watch?v=pl-y9869y0w",
-		// "https://www.youtube.com/watch?v=Uw4NEPE4l3A",
-		// "https://www.youtube.com/watch?v=6tIsT57_nS0",
-		// "https://www.youtube.com/watch?v=2RF0lcTuuYE",
-		// "https://www.youtube.com/watch?v=lymwNQY0dus",
-		// "https://www.youtube.com/watch?v=NTc-I4Z_duc",
-		// "https://www.youtube.com/watch?v=wNSm1TJ84Ac",
-	}
-	for _, u := range urls {
-		d := download.NewDownload(u, configService.Config)
-		d.DownloadProfile = *configService.Config.ProfileCalled("standard video")
-		dm.AddDownload(d)
-		dm.Queue(d)
-	}
 
 	log.Printf("Visit %s for details on installing the bookmarklet and to check status", configService.Config.Server.Address)
 	log.Fatal(srv.ListenAndServe())
@@ -139,7 +123,10 @@ func main() {
 func versionRESTHandler(w http.ResponseWriter, r *http.Request) {
 	if versionInfo.GetInfo().GithubVersionFetched {
 		b, _ := json.Marshal(versionInfo.GetInfo())
-		w.Write(b)
+		_, err := w.Write(b)
+		if err != nil {
+			log.Printf("could not write to client: %s", err)
+		}
 	} else {
 		w.WriteHeader(400)
 	}
@@ -191,7 +178,10 @@ func staticHandler(w http.ResponseWriter, r *http.Request) {
 			return
 		}
 		w.WriteHeader(http.StatusOK)
-		io.Copy(w, f)
+		_, err = io.Copy(w, f)
+		if err != nil {
+			log.Printf("could not write to client: %s", err)
+		}
 		return
 	}
 	w.WriteHeader(http.StatusNotFound)
@@ -227,13 +217,19 @@ func configRESTHandler(w http.ResponseWriter, r *http.Request) {
 			errorRes := errorResponse{Success: false, Error: err.Error()}
 			errorResB, _ := json.Marshal(errorRes)
 			w.WriteHeader(400)
-			w.Write(errorResB)
+			_, err = w.Write(errorResB)
+			if err != nil {
+				log.Printf("could not write to client: %s", err)
+			}
 			return
 		}
 		configService.WriteConfig()
 	}
 	b, _ := json.Marshal(configService.Config)
-	w.Write(b)
+	_, err := w.Write(b)
+	if err != nil {
+		log.Printf("could not write config to client: %s", err)
+	}
 }
 
 func fetchInfoOneRESTHandler(w http.ResponseWriter, r *http.Request) {
@@ -275,7 +271,10 @@ func fetchInfoOneRESTHandler(w http.ResponseWriter, r *http.Request) {
 				errorRes := errorResponse{Success: false, Error: err.Error()}
 				errorResB, _ := json.Marshal(errorRes)
 				w.WriteHeader(400)
-				w.Write(errorResB)
+				_, err = w.Write(errorResB)
+				if err != nil {
+					log.Printf("could not write to client: %s", err)
+				}
 				return
 			}
 
@@ -294,7 +293,10 @@ func fetchInfoOneRESTHandler(w http.ResponseWriter, r *http.Request) {
 
 				succRes := successResponse{Success: true, Message: "download started"}
 				succResB, _ := json.Marshal(succRes)
-				w.Write(succResB)
+				_, err = w.Write(succResB)
+				if err != nil {
+					log.Printf("could not write to client: %s", err)
+				}
 				return
 			}
 
@@ -303,16 +305,16 @@ func fetchInfoOneRESTHandler(w http.ResponseWriter, r *http.Request) {
 				// nil means (probably) that they chose "don't move" - which is fine,
 				// and maps to nil on the Download (the default state).
 				destination := configService.Config.DestinationCalled(thisReq.Destination)
+				dm.ChangeDestination(thisDownload, destination)
 
-				thisDownload.Lock.Lock()
+				//				log.Printf("%#v", thisDownload)
-				thisDownload.Destination = destination
-				thisDownload.Lock.Unlock()
-
-				log.Printf("%#v", thisDownload)
 
 				succRes := successResponse{Success: true, Message: "destination changed"}
 				succResB, _ := json.Marshal(succRes)
-				w.Write(succResB)
+				_, err = w.Write(succResB)
+				if err != nil {
+					log.Printf("could not write to client: %s", err)
+				}
 				return
 			}
 
@@ -321,14 +323,24 @@ func fetchInfoOneRESTHandler(w http.ResponseWriter, r *http.Request) {
 				thisDownload.Stop()
 				succRes := successResponse{Success: true, Message: "download stopped"}
 				succResB, _ := json.Marshal(succRes)
-				w.Write(succResB)
+				_, err = w.Write(succResB)
+				if err != nil {
+					log.Printf("could not write to client: %s", err)
+				}
 				return
 			}
 		}
 
 		// just a get, return the object
+		thisDownload.Lock.Lock()
+		defer thisDownload.Lock.Unlock()
+
 		b, _ := json.Marshal(thisDownload)
-		w.Write(b)
+
+		_, err = w.Write(b)
+		if err != nil {
+			log.Printf("could not write to client: %s", err)
+		}
 		return
 	} else {
 		http.NotFound(w, r)
@@ -337,10 +349,14 @@ func fetchInfoOneRESTHandler(w http.ResponseWriter, r *http.Request) {
 
 func fetchInfoRESTHandler(w http.ResponseWriter, r *http.Request) {
 
-	dm.Lock.Lock()
+	b, err := dm.DownloadsAsJSON()
-	defer dm.Lock.Unlock()
+	if err != nil {
-	b, _ := json.Marshal(dm.Downloads)
+		panic(err)
-	w.Write(b)
+	}
+	_, err = w.Write(b)
+	if err != nil {
+		log.Printf("could not write to client: %s", err)
+	}
 }
 
 func fetchHandler(w http.ResponseWriter, r *http.Request) {
@@ -394,18 +410,14 @@ func fetchHandler(w http.ResponseWriter, r *http.Request) {
 		}
 
 		// create the new download
-		log.Print("creating")
 		newDL := download.NewDownload(url[0], configService.Config)
-		log.Print("adding")
 		dm.AddDownload(newDL)
-		log.Print("done")
 
 		t, err := template.ParseFS(webFS, "web/layout.tmpl", "web/popup.html")
 		if err != nil {
 			panic(err)
 		}
 
-		log.Print("lock dl")
 		newDL.Lock.Lock()
 		defer newDL.Lock.Unlock()
 
@@ -415,6 +427,5 @@ func fetchHandler(w http.ResponseWriter, r *http.Request) {
 		if err != nil {
 			panic(err)
 		}
-		log.Print("unlock dl because rendered")
 	}
 }

version/version.go

@@ -26,7 +26,6 @@ type Manager struct {
 }
 
 func (m *Manager) GetInfo() Info {
-	// log.Print("getting info... b4 lock")
 	m.lock.Lock()
 	defer m.lock.Unlock()
 

web/layout.tmpl

@@ -33,6 +33,9 @@
         .state-downloading {
           color: blue;
         }
+        .state-moved {
+          color: green;
+        }
         .state-complete {
           color: green;
         }

web/popup.html

@@ -34,11 +34,11 @@
         </table>
         <p>You can close this window and your download will continue. Check the <a href="/" target="_gropple_status">Status page</a> to see all downloads in progress.</p>
         {{ if .canStop }}
-        <button x-show="state=='downloading'" class="pure-button" @click="stop()">stop</button>
+        <button x-show="state=='Downloading'" class="pure-button" @click="stop()">stop</button>
         {{ end }}
         <div>
             <h4>Logs</h4>
-            <pre x-text="log">
+            <pre x-text="log" style="height: auto;">
             </pre>
         </div>
     </div>
@@ -101,7 +101,11 @@
                     this.state = info.state;
                     this.playlist_current = info.playlist_current;
                     this.playlist_total = info.playlist_total;
-                    if (this.state != 'choose profile') {
+                    this.destination_chosen = null;
+                    if (info.destination) {
+                        this.destination_chosen = info.destination.name;
+                    }
+                    if (this.state != 'Choose Profile') {
                         this.profile_chosen = true;
                     }
                     this.finished = info.finished;