Update changelog, skip docker for now until it is ready

CHANGELOG.md

@@ -4,16 +4,19 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
-## [v0.6.0] - 2023-03-09
+## [v0.6.0] - 2023-03-15
 
 - Configurable destinations for downloads
+  - Multiple destination directories can be configured
+  - When queueing a download, an alternate destination can be selected
 - When downloading from a playlist, show the total number of videos and how many have been downloaded
 - Show version in web UI
 - Improve index page (show URL of queued downloads instead of nothing)
-- Add docker support
 - Fixes and improvements to capturing output info and showing it in the UI
+- Show all log output in the popup
 - Fixes to handling of queued downloads
 - Fix portable mode to look in binary directory, not current directory
+- Automatically cleanup download list, removing old entries automatically
 
 ## [v0.5.5] - 2022-04-09
 

README.md

@@ -16,9 +16,10 @@ A frontend to youtube-dl (or compatible forks, like yt-dlp) to download videos w
 
 ## Binaries
 
-Binaries are available at https://github.com/tardisx/gropple/releases
+Binaries are available at <https://github.com/tardisx/gropple/releases>
 
-Gropple will automatically check for available updates and prompt you to upgrade.
+Gropple will automatically check for available updates and prompt you to
+upgrade.
 
 ## Running
 
@@ -32,66 +33,75 @@ interface. The address will be printed after startup:
 
 ## Using
 
-Bring up `http://localhost:6283` (or your configured address) in your browser. You 
-should see a link to the bookmarklet at the top of the screen, and the list of
-downloads (currently empty).
+Bring up `http://localhost:6283` (or your configured address) in your browser.
+You should see a link to the bookmarklet at the top of the screen, and the list
+of downloads (currently empty).
 
-Drag the bookmarklet to your favourites bar, or otherwise bookmark it as you 
-see fit. Any kind of browser bookmark should work. The bookmarklet contains
-embedded javascript to pass the URL of whatever page you are currently on back
-to gropple.
+Drag the bookmarklet to your favourites bar, or otherwise bookmark it as you see
+fit. Any kind of browser bookmark should work. The bookmarklet contains embedded
+javascript to pass the URL of whatever page you are currently on back to
+gropple.
 
-So, whenever you are on a page with a video you would like to download just 
-click the bookmarklet.
+Whenever you are on a page with a video you would like to download just click
+the bookmarklet.
 
-A popup window will appear. Choose a download profile and the download will start.
-The status will be shown in the window, updating in real time.
+A popup window will appear. Choose a download profile and the download will
+start. The status will be shown in the window, updating in real time.
 
-You may close this window at any time without stopping the download, the status 
+You may close this window at any time without stopping the download, the status
 of all downloads is available on the index page.
 
 ## Configuration
 
-Click the "config" link on the index page to configure gropple. The default options 
-are fine if you are running on your local machine. If you are running it remotely
-you will need to set the "server address" to ensure the bookmarklet has the correct
-URL in it.
+Click the "config" link on the index page to configure gropple. The default
+options are fine if you are running on your local machine. If you are running it
+remotely you will need to set the "server address" to ensure the bookmarklet has
+the correct URL in it.
 
 ### Configuring Downloaders
 
-Gropple's default configuration uses the original youtube-dl and has two profiles set
-up, one for downloading video, the other for downloading audio (mp3).
+Gropple's default configuration uses `yt-dlp` and has two profiles set up, one
+for downloading video, the other for downloading audio (mp3).
 
-Note that gropple does not include any downloaders, you have to install them separately.
+Note that gropple does not include any downloaders, you have to install them
+separately.
 
-If you would like to use a youtube-dl fork (like [yt-dlp](https://github.com/yt-dlp/yt-dlp))
-or change the options, you can do so on the right hand side. Create as many profiles as you 
-wish, whenever you start a download you can choose the appropriate profile.
+If you would like to use a youtube-dl compatible fork or change the options you
+can do so on the right hand side. Create as many profiles as you wish, whenever
+you start a download you can choose the appropriate profile.
 
-Note that the command arguments must each be specified separately - see the default configuration
-for an example.
+Note that the command arguments must each be specified separately - see the
+default configuration for an example.
 
-While gropple will use your `PATH` to find the executable, you can also specify a full path
-instead. Note that any tools that the downloader calls itself (for instance, ffmpeg) will 
-probably need to be available on your path.
+While gropple will use your `PATH` to find the executable, you can also specify
+a full path instead. Note that any tools that the downloader calls itself (for
+instance, `ffmpeg`) will need to be available on your path.
+
+### Alternate destinations
+
+Gropple supports adding additional optional destinations. By default, all
+downloads will be stored in the main download path specified in the config. You
+can also add one or more destinations, and you can choose one of these
+destinations when queueing a new download, or while it is still downloading from
+the popup.
+
+The file will be moved after downloading is complete.
 
 ## Portable mode
 
-If you'd like to use gropple from a USB stick or similar, copy the config file from
-it's default location (shown when you start gropple) to the same location as the binary, and rename it to `gropple.yml`. 
-
-If that file is present in the same directory as the binary, it will be used instead.
+If you'd like to use gropple from a USB stick or similar, copy the config file
+from its default location (shown when you start gropple) to the same location as
+the binary, and rename it to `gropple.yml`.
 
 ## Problems
 
-Most download problems are probably diagnosable via the log - check in the popup window and scroll
-the log down to the bottom. The most common problem is that youtube-dl cannot be found, or its
-dependency (like ffmpeg) cannot be found on your path. 
+Many download problems are diagnosable via the log - check in the popup window
+and scroll the log down to the bottom. The most common problem is that `yt-dlp`
+cannot be found, or its dependency (like `ffmpeg`) cannot be found on your path.
 
 For other problems, please file an issue on github.
 
 ## TODO
 
-Many things. Please raise an issue after checking the [currently open issues](https://github.com/tardisx/gropple/issues).
-
-
+Many things. Please raise an issue after checking the [currently open
+issues](https://github.com/tardisx/gropple/issues).

build_release.pl

@@ -40,6 +40,7 @@ foreach my $type (keys %build) {
 }
 
 # now docker
+exit 0;
 $ENV{VERSION}="$version";
 system "docker-compose", "-f", "docker-compose.build.yml", "build";
 system "docker", "tag", "tardisx/gropple:$version", "tardisx/gropple:latest";

config/config.go

@@ -57,6 +57,7 @@ type ConfigService struct {
 
 func (cs *ConfigService) LoadTestConfig() {
 	cs.LoadDefaultConfig()
+	cs.Config.Server.DownloadPath = "/tmp"
 	cs.Config.DownloadProfiles = []DownloadProfile{{Name: "test profile", Command: "sleep", Args: []string{"5"}}}
 }
 
@@ -93,7 +94,6 @@ func (cs *ConfigService) LoadDefaultConfig() {
 
 	cs.Config = &defaultConfig
 
-	return
 }
 
 func (c *Config) ProfileCalled(name string) *DownloadProfile {
@@ -318,6 +318,9 @@ func (cs *ConfigService) WriteConfig() {
 	}
 	defer file.Close()
 
-	file.Write(s)
+	_, err = file.Write(s)
+	if err != nil {
+		log.Fatalf("could not write config file %s: %s", path, err)
+	}
 	file.Close()
 }

download/download.go

@@ -1,12 +1,14 @@
 package download
 
 import (
+	"encoding/json"
 	"fmt"
 	"io"
 	"log"
 	"net/url"
 	"os"
 	"os/exec"
+	"path/filepath"
 	"regexp"
 	"strconv"
 	"strings"
@@ -23,7 +25,7 @@ type Download struct {
 	PopupUrl        string                 `json:"popup_url"`
 	Process         *os.Process            `json:"-"`
 	ExitCode        int                    `json:"exit_code"`
-	State           string                 `json:"state"`
+	State           State                  `json:"state"`
 	DownloadProfile config.DownloadProfile `json:"download_profile"`
 	Destination     *config.Destination    `json:"destination"`
 	Finished        bool                   `json:"finished"`
@@ -45,6 +47,32 @@ type Manager struct {
 	Lock         sync.Mutex
 }
 
+func (m *Manager) String() string {
+	m.Lock.Lock()
+	defer m.Lock.Unlock()
+	out := fmt.Sprintf("Max per domain: %d, downloads: %d\n", m.MaxPerDomain, len(m.Downloads))
+
+	for _, dl := range m.Downloads {
+		out = out + fmt.Sprintf("%3d: (%10s) %30s\n", dl.Id, dl.State, dl.Url)
+	}
+
+	return out
+
+}
+
+type State string
+
+const (
+	STATE_PREPARING            State = "Preparing to start"
+	STATE_CHOOSE_PROFILE       State = "Choose Profile"
+	STATE_QUEUED               State = "Queued"
+	STATE_DOWNLOADING          State = "Downloading"
+	STATE_DOWNLOADING_METADATA State = "Downloading metadata"
+	STATE_FAILED               State = "Failed"
+	STATE_COMPLETE             State = "Complete"
+	STATE_MOVED                State = "Moved"
+)
+
 var CanStopDownload = false
 
 var downloadId int32 = 0
@@ -54,13 +82,52 @@ func (m *Manager) ManageQueue() {
 		m.Lock.Lock()
 
 		m.startQueued(m.MaxPerDomain)
-		// m.cleanup()
+		m.moveToDest()
+		m.cleanup()
 		m.Lock.Unlock()
 
 		time.Sleep(time.Second)
 	}
 }
 
+func (m *Manager) DownloadsAsJSON() ([]byte, error) {
+
+	m.Lock.Lock()
+	defer m.Lock.Unlock()
+	for _, dl := range m.Downloads {
+		dl.Lock.Lock()
+		defer dl.Lock.Unlock()
+	}
+	b, err := json.Marshal(m.Downloads)
+	return b, err
+}
+
+func (m *Manager) moveToDest() {
+
+	// move any downloads that are complete and have a dest
+	for _, dl := range m.Downloads {
+
+		dl.Lock.Lock()
+		if dl.Destination != nil && dl.State == STATE_COMPLETE {
+			dl.State = STATE_MOVED
+			for _, fn := range dl.Files {
+				src := filepath.Join(dl.Config.Server.DownloadPath, fn)
+				dst := filepath.Join(dl.Destination.Path, fn)
+				err := os.Rename(src, dst)
+				if err != nil {
+					log.Printf("%s", err)
+					dl.Log = append(dl.Log, fmt.Sprintf("Could not move %s to %s - %s", fn, dl.Destination.Path, err))
+					break
+				} else {
+					dl.Log = append(dl.Log, fmt.Sprintf("Moved %s to %s", fn, dl.Destination.Path))
+
+				}
+			}
+		}
+		dl.Lock.Unlock()
+	}
+}
+
 // startQueued starts any downloads that have been queued, we would not exceed
 // maxRunning. If maxRunning is 0, there is no limit.
 func (m *Manager) startQueued(maxRunning int) {
@@ -70,7 +137,7 @@ func (m *Manager) startQueued(maxRunning int) {
 	for _, dl := range m.Downloads {
 		dl.Lock.Lock()
 
-		if dl.State == "downloading" || dl.State == "preparing to start" {
+		if dl.State == STATE_DOWNLOADING || dl.State == STATE_PREPARING {
 			active[dl.domain()]++
 		}
 		dl.Lock.Unlock()
@@ -81,8 +148,8 @@ func (m *Manager) startQueued(maxRunning int) {
 
 		dl.Lock.Lock()
 
-		if dl.State == "queued" && (maxRunning == 0 || active[dl.domain()] < maxRunning) {
-			dl.State = "preparing to start"
+		if dl.State == STATE_QUEUED && (maxRunning == 0 || active[dl.domain()] < maxRunning) {
+			dl.State = STATE_PREPARING
 			active[dl.domain()]++
 			log.Printf("Starting download for id:%d (%s)", dl.Id, dl.Url)
 
@@ -100,16 +167,17 @@ func (m *Manager) startQueued(maxRunning int) {
 }
 
 // cleanup removes old downloads from the list. Hardcoded to remove them one hour
-// completion.
-func (m *Manager) XXXcleanup() {
+// completion. Expects the Manager to be locked.
+func (m *Manager) cleanup() {
 	newDLs := []*Download{}
 	for _, dl := range m.Downloads {
-
+		dl.Lock.Lock()
 		if dl.Finished && time.Since(dl.FinishedTS) > time.Duration(time.Hour) {
 			// do nothing
 		} else {
 			newDLs = append(newDLs, dl)
 		}
+		dl.Lock.Unlock()
 
 	}
 	m.Downloads = newDLs
@@ -131,7 +199,18 @@ func (m *Manager) GetDlById(id int) (*Download, error) {
 func (m *Manager) Queue(dl *Download) {
 	dl.Lock.Lock()
 	defer dl.Lock.Unlock()
-	dl.State = "queued"
+	dl.State = STATE_QUEUED
+}
+
+func (m *Manager) ChangeDestination(dl *Download, dest *config.Destination) {
+	dl.Lock.Lock()
+	// we can only change destination is certain cases...
+	if dl.State != STATE_FAILED && dl.State != STATE_MOVED {
+		dl.Destination = dest
+	}
+
+	dl.Lock.Unlock()
+
 }
 
 func NewDownload(url string, conf *config.Config) *Download {
@@ -140,7 +219,7 @@ func NewDownload(url string, conf *config.Config) *Download {
 		Id:       int(downloadId),
 		Url:      url,
 		PopupUrl: fmt.Sprintf("/fetch/%d", int(downloadId)),
-		State:    "choose profile",
+		State:    STATE_CHOOSE_PROFILE,
 		Files:    []string{},
 		Log:      []string{},
 		Config:   conf,
@@ -153,7 +232,6 @@ func (m *Manager) AddDownload(dl *Download) {
 	m.Lock.Lock()
 	defer m.Lock.Unlock()
 	m.Downloads = append(m.Downloads, dl)
-	return
 }
 
 // func (dl *Download) AppendLog(text string) {
@@ -173,7 +251,10 @@ func (dl *Download) Stop() {
 	dl.Lock.Lock()
 	defer dl.Lock.Unlock()
 	dl.Log = append(dl.Log, "aborted by user")
-	dl.Process.Kill()
+	err := dl.Process.Kill()
+	if err != nil {
+		log.Printf("could not send kill to process: %s", err)
+	}
 }
 
 // domain returns a domain for this Download. Download should be locked.
@@ -194,7 +275,7 @@ func (dl *Download) domain() string {
 func (dl *Download) Begin() {
 	dl.Lock.Lock()
 
-	dl.State = "downloading"
+	dl.State = STATE_DOWNLOADING
 	cmdSlice := []string{}
 	cmdSlice = append(cmdSlice, dl.DownloadProfile.Args...)
 
@@ -208,7 +289,7 @@ func (dl *Download) Begin() {
 
 	stdout, err := cmd.StdoutPipe()
 	if err != nil {
-		dl.State = "failed"
+		dl.State = STATE_FAILED
 		dl.Finished = true
 		dl.FinishedTS = time.Now()
 		dl.Log = append(dl.Log, fmt.Sprintf("error setting up stdout pipe: %v", err))
@@ -219,7 +300,7 @@ func (dl *Download) Begin() {
 
 	stderr, err := cmd.StderrPipe()
 	if err != nil {
-		dl.State = "failed"
+		dl.State = STATE_FAILED
 		dl.Finished = true
 		dl.FinishedTS = time.Now()
 		dl.Log = append(dl.Log, fmt.Sprintf("error setting up stderr pipe: %v", err))
@@ -231,7 +312,7 @@ func (dl *Download) Begin() {
 	log.Printf("Executing command: %v", cmd)
 	err = cmd.Start()
 	if err != nil {
-		dl.State = "failed"
+		dl.State = STATE_FAILED
 		dl.Finished = true
 		dl.FinishedTS = time.Now()
 		dl.Log = append(dl.Log, fmt.Sprintf("error starting command '%s': %v", dl.DownloadProfile.Command, err))
@@ -258,19 +339,30 @@ func (dl *Download) Begin() {
 	}()
 
 	wg.Wait()
-	cmd.Wait()
 
+	err = cmd.Wait()
 	dl.Lock.Lock()
 
-	log.Printf("Process finished for id: %d (%v)", dl.Id, cmd)
+	if err != nil {
+		log.Printf("process failed for id: %d: %s", dl.Id, err)
 
-	dl.State = "complete"
-	dl.Finished = true
-	dl.FinishedTS = time.Now()
-	dl.ExitCode = cmd.ProcessState.ExitCode()
+		dl.State = STATE_FAILED
+		dl.Finished = true
+		dl.FinishedTS = time.Now()
+		dl.ExitCode = cmd.ProcessState.ExitCode()
 
-	if dl.ExitCode != 0 {
-		dl.State = "failed"
+	} else {
+
+		log.Printf("process finished for id: %d (%v)", dl.Id, cmd)
+
+		dl.State = STATE_COMPLETE
+		dl.Finished = true
+		dl.FinishedTS = time.Now()
+		dl.ExitCode = cmd.ProcessState.ExitCode()
+
+		if dl.ExitCode != 0 {
+			dl.State = STATE_FAILED
+		}
 	}
 	dl.Lock.Unlock()
 
@@ -318,7 +410,7 @@ func (dl *Download) updateMetadata(s string) {
 	matches := etaRE.FindStringSubmatch(s)
 	if len(matches) == 2 {
 		dl.Eta = matches[1]
-		dl.State = "downloading"
+		dl.State = STATE_DOWNLOADING
 
 	}
 
@@ -379,7 +471,7 @@ func (dl *Download) updateMetadata(s string) {
 	metadataDL := regexp.MustCompile(`Downloading JSON metadata page (\d+)`)
 	matches = metadataDL.FindStringSubmatch(s)
 	if len(matches) == 2 {
-		dl.State = "Downloading metadata, page " + matches[1]
+		dl.State = STATE_DOWNLOADING_METADATA
 	}
 
 	// [FixupM3u8] Fixing MPEG-TS in MP4 container of "file [-168849776_456239489].mp4"

download/download_notestdata.go

@@ -0,0 +1,8 @@
+//go:build !testdata
+
+package download
+
+import "github.com/tardisx/gropple/config"
+
+func (m *Manager) AddStressTestData(c *config.ConfigService) {
+}

download/download_test.go

@@ -2,7 +2,11 @@ package download
 
 import (
 	"strings"
+	"sync"
 	"testing"
+	"time"
+
+	"github.com/tardisx/gropple/config"
 )
 
 func TestUpdateMetadata(t *testing.T) {
@@ -76,79 +80,158 @@ func TestUpdateMetadata(t *testing.T) {
 // [download] 100% of 4.64MiB in 00:00
 // [ffmpeg] Merging formats into "Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.mp4"
 
-// func TestQueue(t *testing.T) {
-// 	cs := config.ConfigService{}
-// 	cs.LoadTestConfig()
-// 	conf := cs.Config
+func TestQueue(t *testing.T) {
+	cs := config.ConfigService{}
+	cs.LoadTestConfig()
+	conf := cs.Config
 
-// 	new1 := Download{Id: 1, Url: "http://sub.example.org/foo1", State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
-// 	new2 := Download{Id: 2, Url: "http://sub.example.org/foo2", State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
-// 	new3 := Download{Id: 3, Url: "http://sub.example.org/foo3", State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
-// 	new4 := Download{Id: 4, Url: "http://example.org/", State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
+	new1 := NewDownload("http://sub.example.org/foo1", conf)
+	new2 := NewDownload("http://sub.example.org/foo2", conf)
+	new3 := NewDownload("http://sub.example.org/foo3", conf)
+	new4 := NewDownload("http://example.org/", conf)
 
-// 	dls := Downloads{&new1, &new2, &new3, &new4}
-// 	dls.StartQueued(1)
-// 	time.Sleep(time.Millisecond * 100)
-// 	if dls[0].State == "queued" {
-// 		t.Error("#1 was not started")
-// 	}
-// 	if dls[1].State != "queued" {
-// 		t.Error("#2 is not queued")
-// 	}
-// 	if dls[3].State == "queued" {
-// 		t.Error("#4 is not started")
-// 	}
+	// pretend the user chose a profile for each
+	new1.DownloadProfile = *conf.ProfileCalled("test profile")
+	new2.DownloadProfile = *conf.ProfileCalled("test profile")
+	new3.DownloadProfile = *conf.ProfileCalled("test profile")
+	new4.DownloadProfile = *conf.ProfileCalled("test profile")
+	new1.State = STATE_QUEUED
+	new2.State = STATE_QUEUED
+	new3.State = STATE_QUEUED
+	new4.State = STATE_QUEUED
 
-// 	// this should start no more, as one is still going
-// 	dls.StartQueued(1)
-// 	time.Sleep(time.Millisecond * 100)
-// 	if dls[1].State != "queued" {
-// 		t.Error("#2 was started when it should not be")
-// 	}
+	q := Manager{
+		Downloads:    []*Download{},
+		MaxPerDomain: 2,
+		Lock:         sync.Mutex{},
+	}
 
-// 	dls.StartQueued(2)
-// 	time.Sleep(time.Millisecond * 100)
-// 	if dls[1].State == "queued" {
-// 		t.Error("#2 was not started but it should be")
+	q.AddDownload(new1)
+	q.AddDownload(new2)
+	q.AddDownload(new3)
+	q.AddDownload(new4)
 
-// 	}
+	q.startQueued(1)
 
-// 	dls.StartQueued(2)
-// 	time.Sleep(time.Millisecond * 100)
-// 	if dls[3].State == "queued" {
-// 		t.Error("#4 was not started but it should be")
-// 	}
+	// two should start, one from each of the two domains
+	time.Sleep(time.Millisecond * 100)
+	if q.Downloads[0].State != STATE_DOWNLOADING {
+		t.Errorf("#1 was not downloading - %s instead ", q.Downloads[0].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[1].State != STATE_QUEUED {
+		t.Errorf("#2 is not queued - %s instead", q.Downloads[1].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[2].State != STATE_QUEUED {
+		t.Errorf("#3 is not queued - %s instead", q.Downloads[2].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[3].State != STATE_DOWNLOADING {
+		t.Errorf("#4 is not downloading - %s instead", q.Downloads[3].State)
+		t.Log(q.String())
+	}
 
-// 	// reset them all
-// 	dls[0].State = "queued"
-// 	dls[1].State = "queued"
-// 	dls[2].State = "queued"
-// 	dls[3].State = "queued"
+	// this should start no more, as one is still going
+	q.startQueued(1)
+	time.Sleep(time.Millisecond * 100)
+	if q.Downloads[0].State != STATE_DOWNLOADING {
+		t.Errorf("#1 was not downloading - %s instead ", q.Downloads[0].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[1].State != STATE_QUEUED {
+		t.Errorf("#2 is not queued - %s instead", q.Downloads[1].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[2].State != STATE_QUEUED {
+		t.Errorf("#3 is not queued - %s instead", q.Downloads[2].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[3].State != STATE_DOWNLOADING {
+		t.Errorf("#4 is not downloading - %s instead", q.Downloads[3].State)
+		t.Log(q.String())
+	}
 
-// 	dls.StartQueued(0)
-// 	time.Sleep(time.Millisecond * 100)
+	// wait until the two finish, check
+	time.Sleep(time.Second * 5.0)
+	if q.Downloads[0].State != STATE_COMPLETE {
+		t.Errorf("#1 was not complete - %s instead ", q.Downloads[0].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[1].State != STATE_QUEUED {
+		t.Errorf("#2 is not queued - %s instead", q.Downloads[1].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[2].State != STATE_QUEUED {
+		t.Errorf("#3 is not queued - %s instead", q.Downloads[2].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[3].State != STATE_COMPLETE {
+		t.Errorf("#4 is not complete - %s instead", q.Downloads[3].State)
+		t.Log(q.String())
+	}
 
-// 	// they should all be going
-// 	if dls[0].State == "queued" || dls[1].State == "queued" || dls[2].State == "queued" || dls[3].State == "queued" {
-// 		t.Error("none should be queued")
-// 	}
+	// this should start one more, as one is still going
+	q.startQueued(1)
+	time.Sleep(time.Millisecond * 100)
+	if q.Downloads[0].State != STATE_COMPLETE {
+		t.Errorf("#1 was not complete - %s instead ", q.Downloads[0].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[1].State != STATE_DOWNLOADING {
+		t.Errorf("#2 is not downloading - %s instead", q.Downloads[1].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[2].State != STATE_QUEUED {
+		t.Errorf("#3 is not queued - %s instead", q.Downloads[2].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[3].State != STATE_COMPLETE {
+		t.Errorf("#4 is not complete - %s instead", q.Downloads[3].State)
+		t.Log(q.String())
+	}
 
-// 	// reset them all
-// 	dls[0].State = "queued"
-// 	dls[1].State = "queued"
-// 	dls[2].State = "queued"
-// 	dls[3].State = "queued"
+	// this should start no more, as one is still going
+	q.startQueued(1)
+	time.Sleep(time.Millisecond * 100)
+	if q.Downloads[0].State != STATE_COMPLETE {
+		t.Errorf("#1 was not complete - %s instead ", q.Downloads[0].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[1].State != STATE_DOWNLOADING {
+		t.Errorf("#2 is not downloading - %s instead", q.Downloads[1].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[2].State != STATE_QUEUED {
+		t.Errorf("#3 is not queued - %s instead", q.Downloads[2].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[3].State != STATE_COMPLETE {
+		t.Errorf("#4 is not complete - %s instead", q.Downloads[3].State)
+		t.Log(q.String())
+	}
 
-// 	dls.StartQueued(2)
-// 	time.Sleep(time.Millisecond * 100)
+	// but if we allow two per domain, the other queued one will start
+	q.startQueued(2)
+	time.Sleep(time.Millisecond * 100)
+	if q.Downloads[0].State != STATE_COMPLETE {
+		t.Errorf("#1 was not complete - %s instead ", q.Downloads[0].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[1].State != STATE_DOWNLOADING {
+		t.Errorf("#2 is not downloading - %s instead", q.Downloads[1].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[2].State != STATE_DOWNLOADING {
+		t.Errorf("#3 is not downloading - %s instead", q.Downloads[2].State)
+		t.Log(q.String())
+	}
+	if q.Downloads[3].State != STATE_COMPLETE {
+		t.Errorf("#4 is not complete - %s instead", q.Downloads[3].State)
+		t.Log(q.String())
+	}
 
-// 	// first two should be running, third not (same domain) and 4th running (different domain)
-// 	if dls[0].State == "queued" || dls[1].State == "queued" || dls[2].State != "queued" || dls[3].State == "queued" {
-// 		t.Error("incorrect queued")
-
-// 	}
-
-// }
+}
 
 func TestUpdateMetadataPlaylist(t *testing.T) {
 

download/download_testdata.go

@@ -0,0 +1,28 @@
+//go:build testdata
+
+package download
+
+import "github.com/tardisx/gropple/config"
+
+func (m *Manager) AddStressTestData(c *config.ConfigService) {
+
+	urls := []string{
+		"https://www.youtube.com/watch?v=qG_rRkuGBW8",
+		"https://www.youtube.com/watch?v=ZUzhZpQAU40",
+		"https://www.youtube.com/watch?v=kVxM3eRWGak",
+		"https://www.youtube.com/watch?v=pl-y9869y0w",
+		"https://vimeo.com/783453809",
+		"https://www.youtube.com/watch?v=Uw4NEPE4l3A",
+		"https://www.youtube.com/watch?v=2RF0lcTuuYE",
+		"https://www.youtube.com/watch?v=lymwNQY0dus",
+		"https://www.youtube.com/watch?v=NTc-I4Z_duc",
+		"https://www.youtube.com/watch?v=wNSm1TJ84Ac",
+		"https://vimeo.com/786570322",
+	}
+	for _, u := range urls {
+		d := NewDownload(u, c.Config)
+		d.DownloadProfile = *c.Config.ProfileCalled("standard video")
+		m.AddDownload(d)
+		m.Queue(d)
+	}
+}

go.mod

@@ -4,6 +4,6 @@ go 1.20
 
 require (
 	github.com/gorilla/mux v1.8.0
-	golang.org/x/mod v0.5.1
+	golang.org/x/mod v0.9.0
 	gopkg.in/yaml.v2 v2.4.0
 )

go.sum

@@ -1,7 +1,7 @@
 github.com/gorilla/mux v1.8.0 h1:i40aqfkR1h2SlN9hojwV5ZA91wcXFOvkdNIeFDP5koI=
 github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So=
-golang.org/x/mod v0.5.1 h1:OJxoQ/rynoF0dcCdI7cLPktw/hR2cueqYfjm43oqK38=
-golang.org/x/mod v0.5.1/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro=
+golang.org/x/mod v0.9.0 h1:KENHtAZL2y3NLMYZeHY9DW8HW8V+kQyJsY/V9JlKvCs=
+golang.org/x/mod v0.9.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=

main.go

@@ -21,11 +21,10 @@ import (
 )
 
 var dm *download.Manager
-var downloadId = 0
 var configService *config.ConfigService
 
 var versionInfo = version.Manager{
-	VersionInfo: version.Info{CurrentVersion: "v0.6.0-alpha.1"},
+	VersionInfo: version.Info{CurrentVersion: "v0.6.0-alpha.4"},
 }
 
 //go:embed web
@@ -102,7 +101,10 @@ func main() {
 	// check for a new version every 4 hours
 	go func() {
 		for {
-			versionInfo.UpdateGitHubVersion()
+			err := versionInfo.UpdateGitHubVersion()
+			if err != nil {
+				log.Printf("could not get version info: %s", err)
+			}
 			time.Sleep(time.Hour * 4)
 		}
 	}()
@@ -110,25 +112,7 @@ func main() {
 	// start downloading queued downloads when slots available, and clean up
 	// old entries
 	go dm.ManageQueue()
-
-	urls := []string{
-		"https://www.youtube.com/watch?v=qG_rRkuGBW8",
-		"https://www.youtube.com/watch?v=ZUzhZpQAU40",
-		// "https://www.youtube.com/watch?v=kVxM3eRWGak",
-		// "https://www.youtube.com/watch?v=pl-y9869y0w",
-		// "https://www.youtube.com/watch?v=Uw4NEPE4l3A",
-		// "https://www.youtube.com/watch?v=6tIsT57_nS0",
-		// "https://www.youtube.com/watch?v=2RF0lcTuuYE",
-		// "https://www.youtube.com/watch?v=lymwNQY0dus",
-		// "https://www.youtube.com/watch?v=NTc-I4Z_duc",
-		// "https://www.youtube.com/watch?v=wNSm1TJ84Ac",
-	}
-	for _, u := range urls {
-		d := download.NewDownload(u, configService.Config)
-		d.DownloadProfile = *configService.Config.ProfileCalled("standard video")
-		dm.AddDownload(d)
-		dm.Queue(d)
-	}
+	dm.AddStressTestData(configService)
 
 	log.Printf("Visit %s for details on installing the bookmarklet and to check status", configService.Config.Server.Address)
 	log.Fatal(srv.ListenAndServe())
@@ -139,7 +123,10 @@ func main() {
 func versionRESTHandler(w http.ResponseWriter, r *http.Request) {
 	if versionInfo.GetInfo().GithubVersionFetched {
 		b, _ := json.Marshal(versionInfo.GetInfo())
-		w.Write(b)
+		_, err := w.Write(b)
+		if err != nil {
+			log.Printf("could not write to client: %s", err)
+		}
 	} else {
 		w.WriteHeader(400)
 	}
@@ -191,7 +178,10 @@ func staticHandler(w http.ResponseWriter, r *http.Request) {
 			return
 		}
 		w.WriteHeader(http.StatusOK)
-		io.Copy(w, f)
+		_, err = io.Copy(w, f)
+		if err != nil {
+			log.Printf("could not write to client: %s", err)
+		}
 		return
 	}
 	w.WriteHeader(http.StatusNotFound)
@@ -227,13 +217,19 @@ func configRESTHandler(w http.ResponseWriter, r *http.Request) {
 			errorRes := errorResponse{Success: false, Error: err.Error()}
 			errorResB, _ := json.Marshal(errorRes)
 			w.WriteHeader(400)
-			w.Write(errorResB)
+			_, err = w.Write(errorResB)
+			if err != nil {
+				log.Printf("could not write to client: %s", err)
+			}
 			return
 		}
 		configService.WriteConfig()
 	}
 	b, _ := json.Marshal(configService.Config)
-	w.Write(b)
+	_, err := w.Write(b)
+	if err != nil {
+		log.Printf("could not write config to client: %s", err)
+	}
 }
 
 func fetchInfoOneRESTHandler(w http.ResponseWriter, r *http.Request) {
@@ -275,7 +271,10 @@ func fetchInfoOneRESTHandler(w http.ResponseWriter, r *http.Request) {
 				errorRes := errorResponse{Success: false, Error: err.Error()}
 				errorResB, _ := json.Marshal(errorRes)
 				w.WriteHeader(400)
-				w.Write(errorResB)
+				_, err = w.Write(errorResB)
+				if err != nil {
+					log.Printf("could not write to client: %s", err)
+				}
 				return
 			}
 
@@ -294,7 +293,10 @@ func fetchInfoOneRESTHandler(w http.ResponseWriter, r *http.Request) {
 
 				succRes := successResponse{Success: true, Message: "download started"}
 				succResB, _ := json.Marshal(succRes)
-				w.Write(succResB)
+				_, err = w.Write(succResB)
+				if err != nil {
+					log.Printf("could not write to client: %s", err)
+				}
 				return
 			}
 
@@ -303,16 +305,16 @@ func fetchInfoOneRESTHandler(w http.ResponseWriter, r *http.Request) {
 				// nil means (probably) that they chose "don't move" - which is fine,
 				// and maps to nil on the Download (the default state).
 				destination := configService.Config.DestinationCalled(thisReq.Destination)
+				dm.ChangeDestination(thisDownload, destination)
 
-				thisDownload.Lock.Lock()
-				thisDownload.Destination = destination
-				thisDownload.Lock.Unlock()
-
-				log.Printf("%#v", thisDownload)
+				//				log.Printf("%#v", thisDownload)
 
 				succRes := successResponse{Success: true, Message: "destination changed"}
 				succResB, _ := json.Marshal(succRes)
-				w.Write(succResB)
+				_, err = w.Write(succResB)
+				if err != nil {
+					log.Printf("could not write to client: %s", err)
+				}
 				return
 			}
 
@@ -321,14 +323,24 @@ func fetchInfoOneRESTHandler(w http.ResponseWriter, r *http.Request) {
 				thisDownload.Stop()
 				succRes := successResponse{Success: true, Message: "download stopped"}
 				succResB, _ := json.Marshal(succRes)
-				w.Write(succResB)
+				_, err = w.Write(succResB)
+				if err != nil {
+					log.Printf("could not write to client: %s", err)
+				}
 				return
 			}
 		}
 
 		// just a get, return the object
+		thisDownload.Lock.Lock()
+		defer thisDownload.Lock.Unlock()
+
 		b, _ := json.Marshal(thisDownload)
-		w.Write(b)
+
+		_, err = w.Write(b)
+		if err != nil {
+			log.Printf("could not write to client: %s", err)
+		}
 		return
 	} else {
 		http.NotFound(w, r)
@@ -337,10 +349,14 @@ func fetchInfoOneRESTHandler(w http.ResponseWriter, r *http.Request) {
 
 func fetchInfoRESTHandler(w http.ResponseWriter, r *http.Request) {
 
-	dm.Lock.Lock()
-	defer dm.Lock.Unlock()
-	b, _ := json.Marshal(dm.Downloads)
-	w.Write(b)
+	b, err := dm.DownloadsAsJSON()
+	if err != nil {
+		panic(err)
+	}
+	_, err = w.Write(b)
+	if err != nil {
+		log.Printf("could not write to client: %s", err)
+	}
 }
 
 func fetchHandler(w http.ResponseWriter, r *http.Request) {
@@ -394,18 +410,14 @@ func fetchHandler(w http.ResponseWriter, r *http.Request) {
 		}
 
 		// create the new download
-		log.Print("creating")
 		newDL := download.NewDownload(url[0], configService.Config)
-		log.Print("adding")
 		dm.AddDownload(newDL)
-		log.Print("done")
 
 		t, err := template.ParseFS(webFS, "web/layout.tmpl", "web/popup.html")
 		if err != nil {
 			panic(err)
 		}
 
-		log.Print("lock dl")
 		newDL.Lock.Lock()
 		defer newDL.Lock.Unlock()
 
@@ -415,6 +427,5 @@ func fetchHandler(w http.ResponseWriter, r *http.Request) {
 		if err != nil {
 			panic(err)
 		}
-		log.Print("unlock dl because rendered")
 	}
 }

version/version.go

@@ -26,7 +26,6 @@ type Manager struct {
 }
 
 func (m *Manager) GetInfo() Info {
-	// log.Print("getting info... b4 lock")
 	m.lock.Lock()
 	defer m.lock.Unlock()
 

web/layout.tmpl

@@ -33,6 +33,9 @@
         .state-downloading {
           color: blue;
         }
+        .state-moved {
+          color: green;
+        }
         .state-complete {
           color: green;
         }

web/popup.html

@@ -34,11 +34,11 @@
         </table>
         <p>You can close this window and your download will continue. Check the <a href="/" target="_gropple_status">Status page</a> to see all downloads in progress.</p>
         {{ if .canStop }}
-        <button x-show="state=='downloading'" class="pure-button" @click="stop()">stop</button>
+        <button x-show="state=='Downloading'" class="pure-button" @click="stop()">stop</button>
         {{ end }}
         <div>
             <h4>Logs</h4>
-            <pre x-text="log">
+            <pre x-text="log" style="height: auto;">
             </pre>
         </div>
     </div>
@@ -101,7 +101,11 @@
                     this.state = info.state;
                     this.playlist_current = info.playlist_current;
                     this.playlist_total = info.playlist_total;
-                    if (this.state != 'choose profile') {
+                    this.destination_chosen = null;
+                    if (info.destination) {
+                        this.destination_chosen = info.destination.name;
+                    }
+                    if (this.state != 'Choose Profile') {
                         this.profile_chosen = true;
                     }
                     this.finished = info.finished;