Implement download queue (default size 2) and cleanup old entries after a while

.github/dependabot.yml

@@ -0,0 +1,11 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
+
+version: 2
+updates:
+  - package-ecosystem: "gomod" # See documentation for possible values
+    directory: "/" # Location of package manifests
+    schedule:
+      interval: "daily"

CHANGELOG.md

@@ -0,0 +1,49 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [Unreleased]
+
+## [v0.5.3] - 2021-11-21
+
+- Add config option to limit number of simultaneous downloads
+- Remove old download entries from the index after they are complete
+
+## [v0.5.2] - 2021-10-26
+
+- Provide link to re-display the popup window from the index
+- Visual improvements
+
+## [v0.5.1] - 2021-10-25
+
+- Add note about adblockers potentially blocking the popup
+- Make it possible to refresh the popup window without initiating a new download
+
+## [v0.5.0] - 2021-10-01
+
+- No more command line options, all configuration is now app-managed
+- Beautiful (ok, less ugly) new web interface
+- Multiple youtube-dl profiles, a profile can be chosen for each download
+- Bundled profiles include a standard video download and an mp3 download
+- Configuration via web interface, including download profile configuration
+
+## [v0.4.0] - 2021-09-26
+
+- Moved to semantic versioning
+- Automatic version check, prompts for upgrade in GUI
+- Fixed regex to properly match "merging" lines
+- Automatically refresh index page
+
+## [0.03] - 2021-09-24
+
+- Add option to change command (to use youtube-dlc or other forks) and command line arguments
+- Improve log display in popup
+- Improve documentation (slightly)
+
+## [0.02] - 2021-09-22
+
+- Fix #4 so that deleted files are removed from the results
+
+## [0.01] - 2021-09-22
+
+- Initial release

README.md

@@ -1,12 +1,14 @@
 # gropple
 
-A web service and bookmarklet to download videos with a single click.
+A frontend to youtube-dl (and forks) to download videos with a single click, straight from your web browser.
+
+![Screencast](/screencast.gif)
 
 ## Pre-requisites
 
-* a passing familiarity with the command line
+* some familiarity with the command line
 * youtube-dl (plus any of its required dependencies, like ffmpeg)
-* golang compiler
+* golang compiler (only if you'd like to build from source)
 
 ## Build
 
@@ -16,27 +18,70 @@ A web service and bookmarklet to download videos with a single click.
 
 Binaries are available at https://github.com/tardisx/gropple/releases
 
+Gropple will automatically check for available updates and prompt you to upgrade.
+
 ## Running
 
-    gropple -port 6283 -address http://hostname:6283 -path /downloads
+    ./gropple
 
-With no arguments, it will listen on port 6283 and use an address of 'http://localhost:6283'.
+There are no command line arguments. All configuration is done via the web
+interface. The address will be printed after startup:
 
-The address must be specified so that the bookmarklet can refer to the correct
-host when it is not running on your local machine. You may also need to specify
-a different address if you are running it behind a proxy server or similar.
+    2021/09/30 23:53:00 starting gropple v0.5.0 - https://github.com/tardisx/gropple
+    2021/09/30 23:53:00 go to http://localhost:6123 for details on installing the bookmarklet and to check status
 
 ## Using
 
-Bring up `http://localhost:6283` (or your chosen address) in your browser. You should see a link to the bookmarklet at the top of the screen, and the list of downloads (currently empty).
+Bring up `http://localhost:6283` (or your configured address) in your browser. You 
+should see a link to the bookmarklet at the top of the screen, and the list of
+downloads (currently empty).
 
-Drag the bookmarklet to your favourites bar, or otherwise bookmark it as you see fit.
+Drag the bookmarklet to your favourites bar, or otherwise bookmark it as you 
+see fit. Any kind of browser bookmark should work. The bookmarklet contains
+embedded javascript to pass the URL of whatever page you are currently on back
+to gropple.
 
-Whenever you are on a page with a video you would like to download, simply click the bookmarklet.
+So, whenever you are on a page with a video you would like to download just 
+click the bookmarklet.
 
-A popup window will appear, the download will start on the your gropple server and the status will be shown in the window.
+A popup window will appear. Choose a download profile and the download will start.
+The status will be shown in the window, updating in real time.
 
-You may close this window at any time without stopping the download, the status of all downloads is available on the index page.
+You may close this window at any time without stopping the download, the status 
+of all downloads is available on the index page.
+
+## Configuration
+
+Click the "config" link on the index page to configure gropple. The default options 
+are fine if you are running on your local machine. If you are running it remotely
+you will need to set the "server address" to ensure the bookmarklet has the correct
+URL in it.
+
+### Configuring Downloaders
+
+Gropple's default configuration uses the original youtube-dl and has two profiles set
+up, one for downloading video, the other for downloading audio (mp3).
+
+Note that gropple does not include any downloaders, you have to install them separately.
+
+If you would like to use a youtube-dl fork (like [yt-dlp](https://github.com/yt-dlp/yt-dlp))
+or change the options, you can do so on the right hand side. Create as many profiles as you 
+wish, whenever you start a download you can choose the appropriate profile.
+
+Note that the command arguments must each be specified separately - see the default configuration
+for an example.
+
+While gropple will use your `PATH` to find the executable, you can also specify a full path
+instead. Note that any tools that the downloader calls itself (for instance, ffmpeg) will 
+probably need to be available on your path.
+
+## Problems
+
+Most download problems are probably diagnosable via the log - check in the popup window and scroll
+the log down to the bottom. The most common problem is that youtube-dl cannot be found, or its
+dependency (like ffmpeg) cannot be found on your path. 
+
+For other problems, please file an issue on github.
 
 ## TODO
 

build_release.pl

@@ -7,7 +7,8 @@ open my $fh, "<", "main.go" || die $!;
 
 my $version;
 while (<$fh>) {
-  $version = $1 if /^const\s+currentVersion.*?"(v[\d\.]+)"/;
+  # CurrentVersion: "v0.04"
+  $version = $1 if /CurrentVersion:\s*"(v[\d\.]+)"/;
 }
 close $fh;
 

config/config.go

@@ -0,0 +1,243 @@
+package config
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"log"
+	"os"
+	"strings"
+
+	"gopkg.in/yaml.v2"
+)
+
+type Server struct {
+	Port                   int    `yaml:"port" json:"port"`
+	Address                string `yaml:"address" json:"address"`
+	DownloadPath           string `yaml:"download_path" json:"download_path"`
+	MaximumActiveDownloads int    `yaml:"maximum_active_downloads_per_domain" json:"maximum_active_downloads_per_domain"`
+}
+
+type DownloadProfile struct {
+	Name    string   `yaml:"name" json:"name"`
+	Command string   `yaml:"command" json:"command"`
+	Args    []string `yaml:"args" json:"args"`
+}
+
+type UI struct {
+	PopupWidth  int `yaml:"popup_width" json:"popup_width"`
+	PopupHeight int `yaml:"popup_height" json:"popup_height"`
+}
+
+type Config struct {
+	ConfigVersion    int               `yaml:"config_version" json:"config_version"`
+	Server           Server            `yaml:"server" json:"server"`
+	UI               UI                `yaml:"ui" json:"ui"`
+	DownloadProfiles []DownloadProfile `yaml:"profiles" json:"profiles"`
+}
+
+func TestConfig() *Config {
+	config := DefaultConfig()
+	config.DownloadProfiles = []DownloadProfile{{Name: "test profile", Command: "sleep", Args: []string{"5"}}}
+	return config
+}
+
+func DefaultConfig() *Config {
+	defaultConfig := Config{}
+	stdProfile := DownloadProfile{Name: "standard video", Command: "youtube-dl", Args: []string{
+		"--newline",
+		"--write-info-json",
+		"-f",
+		"bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best",
+	}}
+	mp3Profile := DownloadProfile{Name: "standard mp3", Command: "youtube-dl", Args: []string{
+		"--newline",
+		"--write-info-json",
+		"--extract-audio",
+		"--audio-format", "mp3",
+	}}
+
+	defaultConfig.DownloadProfiles = append(defaultConfig.DownloadProfiles, stdProfile)
+	defaultConfig.DownloadProfiles = append(defaultConfig.DownloadProfiles, mp3Profile)
+
+	defaultConfig.Server.Port = 6123
+	defaultConfig.Server.Address = "http://localhost:6123"
+	defaultConfig.Server.DownloadPath = "./"
+
+	defaultConfig.UI.PopupWidth = 500
+	defaultConfig.UI.PopupHeight = 500
+
+	defaultConfig.Server.MaximumActiveDownloads = 2
+
+	defaultConfig.ConfigVersion = 2
+
+	return &defaultConfig
+}
+
+func (c *Config) ProfileCalled(name string) *DownloadProfile {
+	for _, p := range c.DownloadProfiles {
+		if p.Name == name {
+			return &p
+		}
+	}
+	return nil
+}
+
+func (c *Config) UpdateFromJSON(j []byte) error {
+	newConfig := Config{}
+	err := json.Unmarshal(j, &newConfig)
+	if err != nil {
+		log.Printf("Unmarshal error in config: %v", err)
+		return err
+	}
+
+	// sanity checks
+	if newConfig.UI.PopupHeight < 100 || newConfig.UI.PopupHeight > 2000 {
+		return errors.New("invalid popup height - should be 100-2000")
+	}
+	if newConfig.UI.PopupWidth < 100 || newConfig.UI.PopupWidth > 2000 {
+		return errors.New("invalid popup width - should be 100-2000")
+	}
+
+	// check listen port
+	if newConfig.Server.Port < 1 || newConfig.Server.Port > 65535 {
+		return errors.New("invalid server listen port")
+	}
+
+	// check download path
+	fi, err := os.Stat(newConfig.Server.DownloadPath)
+	if os.IsNotExist(err) {
+		return fmt.Errorf("path '%s' does not exist", newConfig.Server.DownloadPath)
+	}
+	if !fi.IsDir() {
+		return fmt.Errorf("path '%s' is not a directory", newConfig.Server.DownloadPath)
+	}
+
+	if newConfig.Server.MaximumActiveDownloads < 0 {
+		return fmt.Errorf("maximum active downloads can not be < 0")
+	}
+
+	// check profile name uniqueness
+	for i, p1 := range newConfig.DownloadProfiles {
+		for j, p2 := range newConfig.DownloadProfiles {
+			if i != j && p1.Name == p2.Name {
+				return fmt.Errorf("duplicate download profile name '%s'", p1.Name)
+			}
+		}
+	}
+
+	// remove leading/trailing spaces from args and commands and check for emptiness
+	for i := range newConfig.DownloadProfiles {
+		newConfig.DownloadProfiles[i].Name = strings.TrimSpace(newConfig.DownloadProfiles[i].Name)
+
+		if newConfig.DownloadProfiles[i].Name == "" {
+			return errors.New("profile name cannot be empty")
+		}
+
+		newConfig.DownloadProfiles[i].Command = strings.TrimSpace(newConfig.DownloadProfiles[i].Command)
+		if newConfig.DownloadProfiles[i].Command == "" {
+			return fmt.Errorf("command in profile '%s' cannot be empty", newConfig.DownloadProfiles[i].Name)
+		}
+
+		// check the args
+		for j := range newConfig.DownloadProfiles[i].Args {
+			newConfig.DownloadProfiles[i].Args[j] = strings.TrimSpace(newConfig.DownloadProfiles[i].Args[j])
+			if newConfig.DownloadProfiles[i].Args[j] == "" {
+				return fmt.Errorf("argument %d of profile '%s' is empty", j+1, newConfig.DownloadProfiles[i].Name)
+			}
+		}
+
+	}
+
+	*c = newConfig
+	return nil
+}
+
+func configPath() string {
+	dir, err := os.UserConfigDir()
+	if err != nil {
+		log.Fatalf("cannot find a directory to store config: %v", err)
+	}
+	appDir := "gropple"
+
+	fullPath := dir + string(os.PathSeparator) + appDir
+	_, err = os.Stat(fullPath)
+
+	if os.IsNotExist(err) {
+		err := os.Mkdir(fullPath, 0777)
+		if err != nil {
+			log.Fatalf("Could not create config dir '%s': %v", fullPath, err)
+		}
+	}
+
+	fullFilename := fullPath + string(os.PathSeparator) + "config.yml"
+	return fullFilename
+}
+
+func ConfigFileExists() bool {
+	info, err := os.Stat(configPath())
+	if os.IsNotExist(err) {
+		return false
+	}
+	if err != nil {
+		log.Fatal(err)
+	}
+	if info.Size() == 0 {
+		log.Print("config file is 0 bytes?")
+		return false
+	}
+	return true
+}
+
+func LoadConfig() (*Config, error) {
+	path := configPath()
+	b, err := os.ReadFile(path)
+	if err != nil {
+		log.Printf("Could not read config '%s': %v", path, err)
+		return nil, err
+	}
+	c := Config{}
+	err = yaml.Unmarshal(b, &c)
+	if err != nil {
+		log.Printf("Could not parse YAML config '%s': %v", path, err)
+		return nil, err
+	}
+
+	// do migrations
+	configMigrated := false
+	if c.ConfigVersion == 1 {
+		c.Server.MaximumActiveDownloads = 2
+		c.ConfigVersion = 2
+		configMigrated = true
+		log.Print("migrated config from version 1 => 2")
+	}
+
+	if configMigrated {
+		log.Print("Writing new config after version migration")
+		c.WriteConfig()
+	}
+
+	return &c, nil
+}
+
+func (c *Config) WriteConfig() {
+	s, err := yaml.Marshal(c)
+	if err != nil {
+		panic(err)
+	}
+
+	path := configPath()
+	file, err := os.Create(
+		path,
+	)
+
+	if err != nil {
+		log.Fatalf("Could not open config file")
+	}
+	defer file.Close()
+
+	file.Write(s)
+	file.Close()
+
+	log.Printf("Wrote configuration out to %s", path)
+}

download/download.go

@@ -0,0 +1,233 @@
+package download
+
+import (
+	"fmt"
+	"io"
+	"os/exec"
+	"regexp"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/tardisx/gropple/config"
+)
+
+type Download struct {
+	Id              int                    `json:"id"`
+	Url             string                 `json:"url"`
+	PopupUrl        string                 `json:"popup_url"`
+	Pid             int                    `json:"pid"`
+	ExitCode        int                    `json:"exit_code"`
+	State           string                 `json:"state"`
+	DownloadProfile config.DownloadProfile `json:"download_profile"`
+	Finished        bool                   `json:"finished"`
+	FinishedTS      time.Time              `json:"finished_ts"`
+	Files           []string               `json:"files"`
+	Eta             string                 `json:"eta"`
+	Percent         float32                `json:"percent"`
+	Log             []string               `json:"log"`
+	Config          *config.Config
+}
+
+type Downloads []*Download
+
+// StartQueued starts any downloads that have been queued, we would not exceed
+// maxRunning. If maxRunning is 0, there is no limit.
+func (dls Downloads) StartQueued(maxRunning int) {
+	active := 0
+	queued := 0
+
+	for _, dl := range dls {
+		if dl.State == "downloading" {
+			active++
+		}
+		if dl.State == "queued" {
+			queued++
+		}
+	}
+
+	// there is room, so start one
+	if queued > 0 && (active < maxRunning || maxRunning == 0) {
+		for _, dl := range dls {
+			if dl.State == "queued" {
+				dl.State = "downloading"
+				go func() { dl.Begin() }()
+				return
+			}
+		}
+	}
+
+}
+
+// Cleanup removes old downloads from the list. Hardcoded to remove them one hour
+// completion.
+func (dls Downloads) Cleanup() Downloads {
+	newDLs := Downloads{}
+	for _, dl := range dls {
+		if dl.Finished && time.Since(dl.FinishedTS) > time.Duration(time.Hour) {
+			// do nothing
+		} else {
+			newDLs = append(newDLs, dl)
+		}
+	}
+	return newDLs
+}
+
+// Queue queues a download
+func (dl *Download) Queue() {
+	dl.State = "queued"
+}
+
+// Begin starts a download, by starting the command specified in the DownloadProfile.
+// It blocks until the download is complete.
+func (dl *Download) Begin() {
+	dl.State = "downloading"
+	cmdSlice := []string{}
+	cmdSlice = append(cmdSlice, dl.DownloadProfile.Args...)
+
+	// only add the url if it's not empty. This helps us with testing
+	if dl.Url != "" {
+		cmdSlice = append(cmdSlice, dl.Url)
+	}
+
+	cmd := exec.Command(dl.DownloadProfile.Command, cmdSlice...)
+	cmd.Dir = dl.Config.Server.DownloadPath
+
+	stdout, err := cmd.StdoutPipe()
+	if err != nil {
+		dl.State = "failed"
+		dl.Finished = true
+		dl.FinishedTS = time.Now()
+		dl.Log = append(dl.Log, fmt.Sprintf("error setting up stdout pipe: %v", err))
+		return
+	}
+
+	stderr, err := cmd.StderrPipe()
+	if err != nil {
+		dl.State = "failed"
+		dl.Finished = true
+		dl.FinishedTS = time.Now()
+		dl.Log = append(dl.Log, fmt.Sprintf("error setting up stderr pipe: %v", err))
+		return
+	}
+
+	err = cmd.Start()
+	if err != nil {
+		dl.State = "failed"
+		dl.Finished = true
+		dl.FinishedTS = time.Now()
+		dl.Log = append(dl.Log, fmt.Sprintf("error starting command '%s': %v", dl.DownloadProfile.Command, err))
+		return
+	}
+	dl.Pid = cmd.Process.Pid
+
+	var wg sync.WaitGroup
+
+	wg.Add(2)
+	go func() {
+		defer wg.Done()
+		dl.updateDownload(stdout)
+	}()
+
+	go func() {
+		defer wg.Done()
+		dl.updateDownload(stderr)
+	}()
+
+	wg.Wait()
+	cmd.Wait()
+
+	dl.State = "complete"
+	dl.Finished = true
+	dl.FinishedTS = time.Now()
+	dl.ExitCode = cmd.ProcessState.ExitCode()
+
+	if dl.ExitCode != 0 {
+		dl.State = "failed"
+	}
+
+}
+
+func (dl *Download) updateDownload(r io.Reader) {
+	// XXX not sure if we might get a partial line?
+	buf := make([]byte, 1024)
+	for {
+		n, err := r.Read(buf)
+		if n > 0 {
+			s := string(buf[:n])
+			lines := strings.Split(s, "\n")
+
+			for _, l := range lines {
+
+				if l == "" {
+					continue
+				}
+
+				// append the raw log
+				dl.Log = append(dl.Log, l)
+
+				// look for the percent and eta and other metadata
+				dl.updateMetadata(l)
+			}
+		}
+		if err != nil {
+			break
+		}
+	}
+}
+
+func (dl *Download) updateMetadata(s string) {
+
+	// [download]  49.7% of ~15.72MiB at  5.83MiB/s ETA 00:07
+	etaRE := regexp.MustCompile(`download.+ETA +(\d\d:\d\d(?::\d\d)?)$`)
+	matches := etaRE.FindStringSubmatch(s)
+	if len(matches) == 2 {
+		dl.Eta = matches[1]
+		dl.State = "downloading"
+
+	}
+
+	percentRE := regexp.MustCompile(`download.+?([\d\.]+)%`)
+	matches = percentRE.FindStringSubmatch(s)
+	if len(matches) == 2 {
+		p, err := strconv.ParseFloat(matches[1], 32)
+		if err == nil {
+			dl.Percent = float32(p)
+		} else {
+			panic(err)
+		}
+	}
+
+	// This appears once per destination file
+	// [download] Destination: Filename with spaces and other punctuation here be careful!.mp4
+	filename := regexp.MustCompile(`download.+?Destination: (.+)$`)
+	matches = filename.FindStringSubmatch(s)
+	if len(matches) == 2 {
+		dl.Files = append(dl.Files, matches[1])
+	}
+
+	// This means a file has been "created" by merging others
+	// [ffmpeg] Merging formats into "Toto - Africa (Official HD Video)-FTQbiNvZqaY.mp4"
+	mergedFilename := regexp.MustCompile(`Merging formats into "(.+)"$`)
+	matches = mergedFilename.FindStringSubmatch(s)
+	if len(matches) == 2 {
+		dl.Files = append(dl.Files, matches[1])
+	}
+
+	// This means a file has been deleted
+	// Gross - this time it's unquoted and has trailing guff
+	// Deleting original file Toto - Africa (Official HD Video)-FTQbiNvZqaY.f137.mp4 (pass -k to keep)
+	// This is very fragile
+	deletedFile := regexp.MustCompile(`Deleting original file (.+) \(pass -k to keep\)$`)
+	matches = deletedFile.FindStringSubmatch(s)
+	if len(matches) == 2 {
+		// find the index
+		for i, f := range dl.Files {
+			if f == matches[1] {
+				dl.Files = append(dl.Files[:i], dl.Files[i+1:]...)
+				break
+			}
+		}
+	}
+}

download/download_test.go

@@ -0,0 +1,99 @@
+package download
+
+import (
+	"testing"
+
+	"github.com/tardisx/gropple/config"
+)
+
+func TestUpdateMetadata(t *testing.T) {
+	newD := Download{}
+
+	// first time we spot a filename
+	newD.updateMetadata("[download] Destination: Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.f401.mp4")
+	if len(newD.Files) != 1 || newD.Files[0] != "Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.f401.mp4" {
+		t.Fatalf("incorrect Files:%v", newD.Files)
+	}
+
+	// eta's might be xx:xx:xx or xx:xx
+	newD.updateMetadata("[download]   0.0% of 504.09MiB at 135.71KiB/s ETA 01:03:36")
+	if newD.Eta != "01:03:36" {
+		t.Fatalf("bad long eta in dl\n%v", newD)
+	}
+	newD.updateMetadata("[download]   0.0% of 504.09MiB at 397.98KiB/s ETA 21:38")
+	if newD.Eta != "21:38" {
+		t.Fatalf("bad short eta in dl\n%v", newD)
+	}
+
+	// added a new file, now we are tracking two
+	newD.updateMetadata("[download] Destination: Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.f140.m4a")
+	if len(newD.Files) != 2 || newD.Files[1] != "Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.f140.m4a" {
+		t.Fatalf("incorrect Files:%v", newD.Files)
+	}
+
+	// merging
+	newD.updateMetadata("[ffmpeg] Merging formats into \"Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.mp4\"")
+	if len(newD.Files) != 3 || newD.Files[2] != "Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.mp4" {
+		t.Fatalf("did not find merged filename")
+		t.Fatalf("%v", newD.Files)
+	}
+
+	// deletes
+	// TODO. Not sure why I don't always see the "Deleting original file" messages after merge -
+	// maybe a youtube-dl fork thing?
+
+}
+
+// [youtube] wi7Agv1M6PY: Downloading webpage
+// [info] Writing video description metadata as JSON to: Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.info.json
+// [download] Destination: Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.f401.mp4
+// [download]   0.0% of 504.09MiB at 135.71KiB/s ETA 01:03:36
+// [download]   0.0% of 504.09MiB at 397.98KiB/s ETA 21:38
+// [download]   0.0% of 504.09MiB at 918.97KiB/s ETA 09:22
+// [download]   0.0% of 504.09MiB at  1.90MiB/s ETA 04:25
+// ..
+// [download]  99.6% of 504.09MiB at  8.91MiB/s ETA 00:00
+// [download] 100.0% of 504.09MiB at  9.54MiB/s ETA 00:00
+// [download] 100% of 504.09MiB in 01:00
+// [download] Destination: Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.f140.m4a
+// [download]   0.0% of 4.64MiB at 155.26KiB/s ETA 00:30
+// [download]   0.1% of 4.64MiB at 457.64KiB/s ETA 00:10
+// [download]   0.1% of 4.64MiB at  1.03MiB/s ETA 00:04
+// ..
+// [download]  86.2% of 4.64MiB at 10.09MiB/s ETA 00:00
+// [download] 100.0% of 4.64MiB at 10.12MiB/s ETA 00:00
+// [download] 100% of 4.64MiB in 00:00
+// [ffmpeg] Merging formats into "Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.mp4"
+
+// This test is a bit broken, because StartQueued immediately starts the queued
+// download, it
+func TestQueue(t *testing.T) {
+	conf := config.TestConfig()
+
+	new1 := Download{Id: 1, State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
+	new2 := Download{Id: 2, State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
+	new3 := Download{Id: 3, State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
+	new4 := Download{Id: 4, State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
+
+	dls := Downloads{&new1, &new2, &new3, &new4}
+	dls.StartQueued(1)
+	if dls[0].State == "queued" {
+		t.Error("#1 was not started")
+	}
+	if dls[1].State != "queued" {
+		t.Error("#2 is not queued")
+	}
+
+	// this should start no more, as one is still going
+	dls.StartQueued(1)
+	if dls[1].State != "queued" {
+		t.Error("#2 was started when it should not be")
+	}
+
+	dls.StartQueued(2)
+	if dls[1].State == "queued" {
+		t.Error("#2 was not started but it should be")
+
+	}
+
+}

go.mod

@@ -2,4 +2,8 @@ module github.com/tardisx/gropple
 
 go 1.16
 
-require github.com/gorilla/mux v1.8.0
+require (
+	github.com/gorilla/mux v1.8.0
+	golang.org/x/mod v0.5.1
+	gopkg.in/yaml.v2 v2.4.0
+)

go.sum

@@ -1,2 +1,19 @@
 github.com/gorilla/mux v1.8.0 h1:i40aqfkR1h2SlN9hojwV5ZA91wcXFOvkdNIeFDP5koI=
 github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/mod v0.5.1 h1:OJxoQ/rynoF0dcCdI7cLPktw/hR2cueqYfjm43oqK38=
+golang.org/x/mod v0.5.1/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
+gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=

main.go

@@ -3,103 +3,180 @@ package main
 import (
 	"embed"
 	"encoding/json"
-	"flag"
 	"fmt"
 	"html/template"
 	"io"
 	"log"
 	"net/http"
-	"os/exec"
-	"regexp"
 	"strings"
-	"sync"
 	"time"
 
 	"strconv"
 
 	"github.com/gorilla/mux"
+	"github.com/tardisx/gropple/config"
+	"github.com/tardisx/gropple/download"
+	"github.com/tardisx/gropple/version"
 )
 
-type download struct {
-	Id       int      `json:"id"`
-	Url      string   `json:"url"`
-	Pid      int      `json:"pid"`
-	ExitCode int      `json:"exit_code"`
-	State    string   `json:"state"`
-	Finished bool     `json:"finished"`
-	Files    []string `json:"files"`
-	Eta      string   `json:"eta"`
-	Percent  float32  `json:"percent"`
-	Log      []string `json:"log"`
-}
-
-var downloads []*download
+var downloads download.Downloads
 var downloadId = 0
-var downloadPath = "./"
+var conf *config.Config
 
-var address string
-
-const currentVersion = "v0.02"
+var versionInfo = version.Info{CurrentVersion: "v0.5.3"}
 
 //go:embed web
 var webFS embed.FS
 
+type successResponse struct {
+	Success bool   `json:"success"`
+	Message string `json:"message"`
+}
+
+type errorResponse struct {
+	Success bool   `json:"success"`
+	Error   string `json:"error"`
+}
+
 func main() {
-	var port int
-	flag.IntVar(&port, "port", 6283, "port to listen on")
-	flag.StringVar(&address, "address", "http://localhost:6283", "address for the service")
-	flag.StringVar(&downloadPath, "path", "", "path for downloaded files - defaults to current directory")
-	flag.Parse()
+	if !config.ConfigFileExists() {
+		log.Print("No config file - creating default config")
+		conf = config.DefaultConfig()
+		conf.WriteConfig()
+	} else {
+		loadedConfig, err := config.LoadConfig()
+		if err != nil {
+			log.Fatal(err)
+		}
+		conf = loadedConfig
+	}
 
 	r := mux.NewRouter()
-	r.HandleFunc("/", HomeHandler)
-	r.HandleFunc("/fetch", FetchHandler)
-	r.HandleFunc("/fetch/info/{id}", FetchInfoHandler)
+	r.HandleFunc("/", homeHandler)
+	r.HandleFunc("/config", configHandler)
+	r.HandleFunc("/fetch", fetchHandler)
+	r.HandleFunc("/fetch/{id}", fetchHandler)
+
+	// info for the list
+	r.HandleFunc("/rest/fetch", fetchInfoRESTHandler)
+	// info for one, including update
+	r.HandleFunc("/rest/fetch/{id}", fetchInfoOneRESTHandler)
+	r.HandleFunc("/rest/version", versionRESTHandler)
+	r.HandleFunc("/rest/config", configRESTHandler)
 
 	http.Handle("/", r)
 
 	srv := &http.Server{
 		Handler: r,
-		Addr:    fmt.Sprintf(":%d", port),
+		Addr:    fmt.Sprintf(":%d", conf.Server.Port),
 		// Good practice: enforce timeouts for servers you create!
 		WriteTimeout: 5 * time.Second,
 		ReadTimeout:  5 * time.Second,
 	}
 
-	log.Printf("starting gropple %s - https://github.com/tardisx/gropple", currentVersion)
-	log.Printf("go to %s for details on installing the bookmarklet and to check status", address)
-	log.Fatal(srv.ListenAndServe())
+	// check for a new version every 4 hours
+	go func() {
+		for {
+			versionInfo.UpdateGitHubVersion()
+			time.Sleep(time.Hour * 4)
+		}
+	}()
 
+	// start downloading queued downloads when slots available, and clean up
+	// old entries
+	go func() {
+		for {
+			downloads.StartQueued(conf.Server.MaximumActiveDownloads)
+			downloads = downloads.Cleanup()
+			time.Sleep(time.Second)
+		}
+	}()
+
+	log.Printf("starting gropple %s - https://github.com/tardisx/gropple", versionInfo.CurrentVersion)
+	log.Printf("go to %s for details on installing the bookmarklet and to check status", conf.Server.Address)
+	log.Fatal(srv.ListenAndServe())
 }
 
-func HomeHandler(w http.ResponseWriter, r *http.Request) {
+// versionRESTHandler returns the version information, if we have up-to-date info from github
+func versionRESTHandler(w http.ResponseWriter, r *http.Request) {
+	if versionInfo.GithubVersionFetched {
+		b, _ := json.Marshal(versionInfo)
+		w.Write(b)
+	} else {
+		w.WriteHeader(400)
+	}
+}
+
+// homeHandler returns the main index page
+func homeHandler(w http.ResponseWriter, r *http.Request) {
 	w.WriteHeader(http.StatusOK)
 
-	bookmarkletURL := fmt.Sprintf("javascript:(function(f,s,n,o){window.open(f+encodeURIComponent(s),n,o)}('%s/fetch?url=',window.location,'yourform','width=500,height=500'));", address)
+	bookmarkletURL := fmt.Sprintf("javascript:(function(f,s,n,o){window.open(f+encodeURIComponent(s),n,o)}('%s/fetch?url=',window.location,'yourform','width=%d,height=%d'));", conf.Server.Address, conf.UI.PopupWidth, conf.UI.PopupHeight)
 
-	t, err := template.ParseFS(webFS, "web/layout.tmpl", "web/index.html")
+	t, err := template.ParseFS(webFS, "web/layout.tmpl", "web/menu.tmpl", "web/index.html")
 	if err != nil {
 		panic(err)
 	}
 
 	type Info struct {
-		Downloads      []*download
+		Downloads      []*download.Download
 		BookmarkletURL template.URL
+		Config         *config.Config
 	}
 
 	info := Info{
 		Downloads:      downloads,
 		BookmarkletURL: template.URL(bookmarkletURL),
+		Config:         conf,
 	}
 
 	err = t.ExecuteTemplate(w, "layout", info)
 	if err != nil {
 		panic(err)
 	}
-
 }
 
-func FetchInfoHandler(w http.ResponseWriter, r *http.Request) {
+// configHandler returns the configuration page
+func configHandler(w http.ResponseWriter, r *http.Request) {
+	w.WriteHeader(http.StatusOK)
+
+	t, err := template.ParseFS(webFS, "web/layout.tmpl", "web/menu.tmpl", "web/config.html")
+	if err != nil {
+		panic(err)
+	}
+
+	err = t.ExecuteTemplate(w, "layout", nil)
+	if err != nil {
+		panic(err)
+	}
+}
+
+// configRESTHandler handles both reading and writing of the configuration
+func configRESTHandler(w http.ResponseWriter, r *http.Request) {
+
+	if r.Method == "POST" {
+		log.Printf("Updating config")
+		b, err := io.ReadAll(r.Body)
+		if err != nil {
+			panic(err)
+		}
+		err = conf.UpdateFromJSON(b)
+
+		if err != nil {
+			errorRes := errorResponse{Success: false, Error: err.Error()}
+			errorResB, _ := json.Marshal(errorRes)
+			w.WriteHeader(400)
+			w.Write(errorResB)
+			return
+		}
+		conf.WriteConfig()
+	}
+	b, _ := json.Marshal(conf)
+	w.Write(b)
+}
+
+//
+func fetchInfoOneRESTHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	idString := vars["id"]
 	if idString != "" {
@@ -109,34 +186,126 @@ func FetchInfoHandler(w http.ResponseWriter, r *http.Request) {
 			return
 		}
 
+		// find the download
+		var thisDownload *download.Download
 		for _, dl := range downloads {
 			if dl.Id == id {
-				b, _ := json.Marshal(dl)
-				w.Write(b)
+				thisDownload = dl
+			}
+		}
+		if thisDownload == nil {
+			http.NotFound(w, r)
+			return
+		}
+
+		if r.Method == "POST" {
+
+			type updateRequest struct {
+				Action  string `json:"action"`
+				Profile string `json:"profile"`
+			}
+
+			thisReq := updateRequest{}
+
+			b, err := io.ReadAll(r.Body)
+			if err != nil {
+				panic(err)
+			}
+
+			err = json.Unmarshal(b, &thisReq)
+			if err != nil {
+				errorRes := errorResponse{Success: false, Error: err.Error()}
+				errorResB, _ := json.Marshal(errorRes)
+				w.WriteHeader(400)
+				w.Write(errorResB)
+				return
+			}
+
+			if thisReq.Action == "start" {
+				// find the profile they asked for
+				profile := conf.ProfileCalled(thisReq.Profile)
+				if profile == nil {
+					panic("bad profile name?")
+				}
+				// set the profile
+				thisDownload.DownloadProfile = *profile
+
+				thisDownload.Queue()
+				succRes := successResponse{Success: true, Message: "download started"}
+				succResB, _ := json.Marshal(succRes)
+				w.Write(succResB)
 				return
 			}
 		}
+
+		// just a get, return the object
+		b, _ := json.Marshal(thisDownload)
+		w.Write(b)
+		return
 	} else {
 		http.NotFound(w, r)
 	}
 }
 
-func FetchHandler(w http.ResponseWriter, r *http.Request) {
+func fetchInfoRESTHandler(w http.ResponseWriter, r *http.Request) {
+	b, _ := json.Marshal(downloads)
+	w.Write(b)
+}
+
+func fetchHandler(w http.ResponseWriter, r *http.Request) {
+
+	// if they refreshed the popup, just load the existing object, don't
+	// create a new one
+	vars := mux.Vars(r)
+	idString := vars["id"]
+
+	idInt, err := strconv.ParseInt(idString, 10, 32)
+	if err == nil && idInt > 0 {
+		for _, dl := range downloads {
+			if dl.Id == int(idInt) {
+				t, err := template.ParseFS(webFS, "web/layout.tmpl", "web/popup.html")
+				if err != nil {
+					panic(err)
+				}
+
+				templateData := map[string]interface{}{"dl": dl, "config": conf}
+
+				err = t.ExecuteTemplate(w, "layout", templateData)
+				if err != nil {
+					panic(err)
+				}
+				return
+			}
+		}
+
+	}
 
 	query := r.URL.Query()
-	url, present := query["url"] //filters=["color", "price", "brand"]
+	url, present := query["url"]
 
 	if !present {
-		fmt.Fprint(w, "something")
+		w.WriteHeader(400)
+		fmt.Fprint(w, "No url supplied")
+		return
 	} else {
 
+		// check the URL for a sudden but inevitable betrayal
+		if strings.Contains(url[0], conf.Server.Address) {
+			w.WriteHeader(400)
+			fmt.Fprint(w, "you musn't gropple your gropple :-)")
+			return
+		}
+
 		// create the record
 		// XXX should be atomic!
 		downloadId++
-		newDownload := download{
+		newDownload := download.Download{
+			Config: conf,
+
 			Id:       downloadId,
 			Url:      url[0],
-			State:    "starting",
+			PopupUrl: fmt.Sprintf("/fetch/%d", downloadId),
+			State:    "choose profile",
 			Finished: false,
 			Eta:      "?",
 			Percent:  0.0,
@@ -147,163 +316,20 @@ func FetchHandler(w http.ResponseWriter, r *http.Request) {
 
 		newDownload.Log = append(newDownload.Log, "start of log...")
 
-		go func() {
-			queue(&newDownload)
-		}()
+		// go func() {
+		// 	newDownload.Begin()
+		// }()
+
 		t, err := template.ParseFS(webFS, "web/layout.tmpl", "web/popup.html")
 		if err != nil {
 			panic(err)
 		}
-		err = t.ExecuteTemplate(w, "layout", newDownload)
+
+		templateData := map[string]interface{}{"dl": newDownload, "config": conf}
+
+		err = t.ExecuteTemplate(w, "layout", templateData)
 		if err != nil {
 			panic(err)
 		}
-
-		//		fmt.Fprintf(w, "Started DL %d!", downloadId)
 	}
 }
-
-func queue(dl *download) {
-
-	cmd := exec.Command(
-		"youtube-dl",
-		"--write-info-json",
-		"-f", "bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best",
-		"--newline", dl.Url,
-	)
-	cmd.Dir = downloadPath
-
-	stdout, err := cmd.StdoutPipe()
-	if err != nil {
-		dl.State = "failed"
-		dl.Finished = true
-		dl.Log = append(dl.Log, fmt.Sprintf("error setting up stdout pipe: %v", err))
-		return
-	}
-
-	stderr, err := cmd.StderrPipe()
-	if err != nil {
-		dl.State = "failed"
-		dl.Finished = true
-		dl.Log = append(dl.Log, fmt.Sprintf("error setting up stderr pipe: %v", err))
-		return
-	}
-
-	err = cmd.Start()
-	if err != nil {
-		dl.State = "failed"
-		dl.Finished = true
-		dl.Log = append(dl.Log, fmt.Sprintf("error starting youtube-dl: %v", err))
-		return
-	}
-	dl.Pid = cmd.Process.Pid
-
-	var wg sync.WaitGroup
-
-	wg.Add(2)
-	go func() {
-		defer wg.Done()
-		updateDownload(stdout, dl)
-	}()
-
-	go func() {
-		defer wg.Done()
-		updateDownload(stderr, dl)
-	}()
-
-	wg.Wait()
-	cmd.Wait()
-
-	dl.State = "complete"
-	dl.Finished = true
-	dl.ExitCode = cmd.ProcessState.ExitCode()
-
-	if dl.ExitCode != 0 {
-		dl.State = "failed"
-	}
-
-}
-
-func updateDownload(r io.Reader, dl *download) {
-	// XXX not sure if we might get a partial line?
-	buf := make([]byte, 1024)
-	for {
-		n, err := r.Read(buf)
-		if n > 0 {
-			s := string(buf[:n])
-			lines := strings.Split(s, "\n")
-
-			for _, l := range lines {
-
-				if l == "" {
-					continue
-				}
-
-				// append the raw log
-				dl.Log = append(dl.Log, l)
-
-				// look for the percent and eta and other metadata
-				updateMetadata(dl, l)
-			}
-		}
-		if err != nil {
-			break
-		}
-	}
-}
-
-func updateMetadata(dl *download, s string) {
-
-	// [download]  49.7% of ~15.72MiB at  5.83MiB/s ETA 00:07
-	etaRE := regexp.MustCompile(`download.+ETA +(\d\d:\d\d)`)
-	matches := etaRE.FindStringSubmatch(s)
-	if len(matches) == 2 {
-		dl.Eta = matches[1]
-		dl.State = "downloading"
-
-	}
-
-	percentRE := regexp.MustCompile(`download.+?([\d\.]+)%`)
-	matches = percentRE.FindStringSubmatch(s)
-	if len(matches) == 2 {
-		p, err := strconv.ParseFloat(matches[1], 32)
-		if err == nil {
-			dl.Percent = float32(p)
-		} else {
-			panic(err)
-		}
-	}
-
-	// This appears once per destination file
-	// [download] Destination: Filename with spaces and other punctuation here be careful!.mp4
-	filename := regexp.MustCompile(`download.+?Destination: (.+)$`)
-	matches = filename.FindStringSubmatch(s)
-	if len(matches) == 2 {
-		dl.Files = append(dl.Files, matches[1])
-	}
-
-	// This means a file has been "created" by merging others
-	// [ffmpeg] Merging formats into "Toto - Africa (Official HD Video)-FTQbiNvZqaY.mp4"
-	mergedFilename := regexp.MustCompile(`Merging formats into "(.+)$`)
-	matches = mergedFilename.FindStringSubmatch(s)
-	if len(matches) == 2 {
-		dl.Files = append(dl.Files, matches[1])
-	}
-
-	// This means a file has been deleted
-	// Gross - this time it's unquoted and has trailing guff
-	// Deleting original file Toto - Africa (Official HD Video)-FTQbiNvZqaY.f137.mp4 (pass -k to keep)
-	// This is very fragile
-	deletedFile := regexp.MustCompile(`Deleting original file (.+) \(pass -k to keep\)$`)
-	matches = deletedFile.FindStringSubmatch(s)
-	if len(matches) == 2 {
-		// find the index
-		for i, f := range dl.Files {
-			if f == matches[1] {
-				dl.Files = append(dl.Files[:i], dl.Files[i+1:]...)
-				break
-			}
-		}
-	}
-
-}

version/version.go

@@ -0,0 +1,80 @@
+// Package version deals with versioning of the software
+package version
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log"
+	"net/http"
+
+	"golang.org/x/mod/semver"
+)
+
+type Info struct {
+	CurrentVersion       string `json:"current_version"`
+	GithubVersion        string `json:"github_version"`
+	UpgradeAvailable     bool   `json:"upgrade_available"`
+	GithubVersionFetched bool   `json:"-"`
+}
+
+func (i *Info) UpdateGitHubVersion() error {
+	i.GithubVersionFetched = false
+	versionUrl := "https://api.github.com/repos/tardisx/gropple/releases"
+	resp, err := http.Get(versionUrl)
+	if err != nil {
+		log.Printf("Error getting response: %v", err)
+		return err
+	}
+	defer resp.Body.Close()
+
+	b, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return fmt.Errorf("failed to read body: %v", err)
+	}
+
+	type release struct {
+		HTMLUrl string `json:"html_url"`
+		TagName string `json:"tag_name"`
+		Name    string `json:"name"`
+	}
+
+	var releases []release
+
+	err = json.Unmarshal(b, &releases)
+	if err != nil {
+		return fmt.Errorf("failed to read unmarshal: %v", err)
+	}
+	if len(releases) == 0 {
+		log.Printf("found no releases on github?")
+		return errors.New("no releases found")
+	}
+
+	i.GithubVersion = releases[0].Name
+
+	i.GithubVersionFetched = true
+	i.UpgradeAvailable = i.canUpgrade()
+	return nil
+}
+
+func (i *Info) canUpgrade() bool {
+	if !i.GithubVersionFetched {
+		return false
+	}
+
+	log.Printf("We are %s, github is %s", i.CurrentVersion, i.GithubVersion)
+
+	if !semver.IsValid(i.CurrentVersion) {
+		log.Printf("current version %s is invalid", i.CurrentVersion)
+	}
+
+	if !semver.IsValid(i.GithubVersion) {
+		log.Printf("github version %s is invalid", i.GithubVersion)
+	}
+
+	if semver.Compare(i.CurrentVersion, i.GithubVersion) == -1 {
+		return true
+	}
+	return false
+}

web/config.html

@@ -0,0 +1,164 @@
+{{ define "content" }}
+
+{{ template "menu.tmpl" . }}
+
+<div x-data="config()" x-init="fetch_config();">
+
+    <p class="error"  x-show="error_message"  x-transition.duration.500ms x-text="error_message"></p>
+    <p class="success" x-show="success_message" x-transition.duration.500ms x-text="success_message"></p>
+
+    <p>Note: changes are not saved until the "Save Config" button is pressed at the bottom of the page.</p>
+
+    <div class="pure-g">
+
+        <div class="pure-u-md-1-2 pure-u-1">
+   
+            <form class="pure-form pure-form-stacked gropple-config">
+                <fieldset>
+
+                    <legend>Server</legend>
+
+                    <label for="config-server-port">Listen Port</label>
+                    <input type="text" id="config-server-port" placeholder="port number" x-model.number="config.server.port" />
+                    <span class="pure-form-message">The port the web server will listen on.</span>
+
+                    <label for="config-server-address">Server address (URL)</label>
+                    <input type="text" id="config-server-address" class="input-long" placeholder="server address" x-model="config.server.address" />
+                    <span class="pure-form-message">
+                        The address the service will be available on. Generally it will be http://hostname:port where
+                        hostname is the host the server is running on, and port is the port you set above.
+                    </span>
+
+                    <label for="config-server-downloadpath">Download path</label>
+                    <input type="text" id="config-server-downloadpath" placeholder="path" class="input-long" x-model="config.server.download_path" />
+                    <span class="pure-form-message">The path on the server to download files to.</span>
+
+                    <label for="config-server-max-downloads">Maximum active downloads</label>
+                    <input type="text" id="config-server-max-downloads" placeholder="2" class="input-long" x-model.number="config.server.maximum_active_downloads_per_domain" />
+                    <span class="pure-form-message">How many downloads can be simultaneously active. Use '0' for no limit.</span>
+
+                    <legend>UI</legend>
+
+                    <p>Note that changes to the popup dimensions will require you to recreate your bookmarklet.</p>
+
+                    <label for="config-ui-popupwidth">Popup Width</label>
+                    <input type="text" id="config-ui-popupwidth" placeholder="width in pixels" x-model.number="config.ui.popup_width" />
+                    <span class="pure-form-message">The width of popup windows in pixels.</span>
+
+                    <label for="config-ui-popupheight">Popup Height</label>
+                    <input type="text" id="config-ui-popupheight" placeholder="height in pixels" x-model.number="config.ui.popup_height" />
+                    <span class="pure-form-message">The height of popup windows in pixels.</span>
+
+                </fieldset>
+            </form>
+
+        </div>
+        <div class="pure-u-md-1-2 pure-u-1">
+            <form class="pure-form gropple-config">
+                <fieldset>
+
+                    <legend>Download Profiles</legend>
+
+                    <p>Gropple supports multiple download profiles. Each profile specifies a different youtube-dl 
+                        compatible command, and arguments. When starting a download, you may choose which profile 
+                        to use. The URL will be appended to the argument list at the end.
+                    </p>
+
+                    <hr>
+
+                    <template x-for="(profile, i) in config.profiles">
+                        <div>
+                            <label x-bind:for="'config-profiles-'+i+'-name'">Name of profile <span x-text="i+1"></span>
+                            </label>                                
+                            
+                            <input type="text" x-bind:id="'config-profiles-'+i+'-name'" class="input-long" placeholder="name" x-model="profile.name" />
+                            <button class="pure-button button-del" href="#" @click.prevent="config.profiles.splice(i, 1);;">delete profile</button>
+
+                            <span class="pure-form-message">The name of this profile. For your information only.</span>
+
+                            <label x-bind:for="'config-profiles-'+i+'-command'">Command to run</label>
+                            <input type="text" x-bind:id="'config-profiles-'+i+'-command'" class="input-long" placeholder="name" x-model="profile.command" />
+                            <span class="pure-form-message">Which command to run. Your path will be searched, or you can specify the full path here.</span>
+
+
+                            <label>Arguments</label>
+
+                            <template x-for="(arg, j) in profile.args">
+                                <div>
+                                    <input type="text" x-bind:id="'config-profiles-'+i+'-arg-'+j" placeholder="arg" x-model="profile.args[j]" />
+                                    <button class="pure-button button-del" href="#" @click.prevent="profile.args.splice(j, 1);;">delete arg</button>
+                                </div>
+                            </template>
+
+                            <button class="pure-button button-add" href="#" @click.prevent="profile.args.push('');">add arg</button>
+
+                            <hr>
+
+                        </div>
+                    </template>
+
+                    <button class="pure-button button-add" href="#" @click.prevent="config.profiles.push({name: 'new profile', command: 'youtube-dl', args: []});">add profile</button>
+
+                </fieldset>
+            </form>
+        </div>
+    </div>
+    <div class="pure-g">
+        <div class="pure-u-1">
+            <button class="pure-button pure-button-primary" @click="save_config();" href="#">Save Config</button>
+        </div>
+    </div>
+
+
+</div>
+
+{{ end }}
+
+{{ define "js" }}
+<script>
+    function config() {
+        return { 
+            config: { server : {}, ui : {}, profiles: [] }, 
+            error_message: '',
+            success_message: '',
+
+            fetch_config() {
+                fetch('/rest/config')
+                .then(response => response.json())
+                .then(config => {
+                    this.config = config;
+                })
+                .catch(error => {
+                    console.log('failed to fetch config', error);
+                });
+            },
+            save_config() {
+                let op = {
+                   method: 'POST',
+                   body: JSON.stringify(this.config),
+                   headers: { 'Content-Type': 'application/json' }
+                };
+                fetch('/rest/config', op)
+                .then(response => {
+                    return response.json();
+                })
+                .then(response => {
+                    if (response.error) {
+                        this.error_message = response.error;
+                        this.success_message = '';
+                        document.body.scrollTop = document.documentElement.scrollTop = 0;
+                    } else {
+                        this.error_message = '';
+                        this.success_message = 'configuration saved';
+                        document.body.scrollTop = document.documentElement.scrollTop = 0;
+                        this.config = response;
+                    }
+                })
+                .catch(error => {
+                    console.log('exception' ,error);
+                });
+            }
+        }
+    }
+</script>
+{{ end }}

web/index.html

@@ -1,29 +1,49 @@
 {{ define "content" }}
-<div>
-<p>
-    Drag this bookmarklet: <a href="{{ .BookmarkletURL }}">Gropple</a> to your bookmark bar, and click it
-    on any page you want to grab the video from.
-</p>
-</div>
 
-<div>
+{{ template "menu.tmpl" . }}
+
+<div x-data="index()" x-init="fetch_data(); fetch_version()">
+
+    <p x-cloak x-show="version && version.upgrade_available">
+        <a href="https://github.com/tardisx/gropple/releases">Upgrade is available</a> - 
+        you have 
+        <span x-text="version.current_version"></span> and 
+        <span x-text="version.github_version"></span>
+    is available.</p>
+
+    <div>
+        <p>
+            Drag this bookmarklet: <a href="{{ .BookmarkletURL }}">Gropple</a> to your bookmark bar, and click it
+            on any page you want to grab the video from.
+        </p>
+	<p>
+	    Please note that some adblockers may prevent the bookmarklet from opening the popup window.
+	</p>
+    </div>
+
     <table class="pure-table">
         <thead>
             <tr>
-                <th>id</th><th>filename</th><th>url</th><th>state</th><th>percent</th><th>eta</th><th>finished</th>
+		    <th>id</th><th>filename</th><th>url</th><th>show</th><th>state</th><th>percent</th><th>eta</th><th>finished</th>
             </tr>
         </thead>
         <tbody>
+            <template x-for="item in items">
+                <tr>
+                    <td x-text="item.id"></td>
+                    <td x-text="item.files"></td>
+                    <td><a class="int-link" x-bind:href="item.url">&#x2197;</a></td>
+                    <td><a class="int-link" @click="show_popup(item)" href="#">&#x1F4C4;</a></td>
+                    <td :class="'state-'+item.state" x-text="item.state"></td>
+                    <td x-text="item.percent"></td>
+                    <td x-text="item.eta"></td>
+                    <td x-text="item.finished ? '&#x2714;' : '-'"></td>
+                </tr>
+    
+            </template>
+              
+
             {{ range $k, $v := .Downloads }}
-            <tr>
-                <td>{{ $v.Id }}</td>
-                <td>{{ range $_, $f := $v.Files }}{{ $f }}<br>{{ end }}</td>
-                <td><a href="{{ $v.Url }}">link</a></td>
-                <td>{{ $v.State }}</td>
-                <td>{{ $v.Percent }}</td>
-                <td>{{ $v.Eta }}</td>
-                <td>{{ $v.Finished }}</td>
-            </tr>
             {{ end }}
         </tbody>  
     </table>
@@ -31,4 +51,37 @@
 {{ end }}
 
 {{ define "js" }}
-{{ end }}
+<script>
+    function index() {
+        return { 
+            items: [], version: {},
+            fetch_version() {
+                fetch('/rest/version')
+                .then(response => response.json())
+                .then(info => {
+                    this.version = info;
+                    setTimeout(() => { this.fetch_version() }, 1000 * 60 );
+                })
+                .catch(error => {
+                    console.log('failed to fetch version info - will retry');
+                    setTimeout(() => { this.fetch_version() }, 1000 );
+                });
+            },
+            fetch_data() {
+                fetch('/rest/fetch')
+                .then(response => response.json())
+                .then(info => {
+                    // will be null if no downloads yet
+                    if (info) {
+                        this.items = info;
+                    }
+                    setTimeout(() => { this.fetch_data() }, 1000);
+                })
+            },
+            show_popup(item) {
+			    window.open(item.popup_url, item.id, "width={{ .Config.UI.PopupWidth }},height={{ .Config.UI.PopupHeight }}");
+	    },
+        }
+    }
+</script>
+{{ end }}

web/layout.tmpl

@@ -8,12 +8,65 @@
     <link rel="stylesheet" href="https://unpkg.com/purecss@2.0.6/build/pure-min.css" integrity="sha384-Uu6IeWbM+gzNVXJcM9XV3SohHtmWE+3VGi496jvgX1jyvDTXfdK+rfZc8C1Aehk5" crossorigin="anonymous">
     <link rel="stylesheet" href="https://unpkg.com/purecss@2.0.6/build/grids-responsive-min.css">
     <style>
-        pre { font-size: 70%; }
+        pre {
+          font-size: 60%;
+          height: 100px;
+          overflow:auto;
+        }
+        footer {
+          padding-top: 50px;
+          font-size: 30%;
+        }
+        .int-link {
+          text-decoration: none;
+          hover { color: red; }
+        }
+        .state-failed {
+          color: red;
+        }
+        .state-downloading {
+          color: blue;
+        }
+        .state-complete {
+          color: green;
+        }
+        .gropple-config {
+          font-size: 80%;
+        }
+        .gropple-config input.input-long {
+          width: 27em;
+        }
+        .gropple-config button {
+          border-radius: 12px;
+        }
+        .gropple-config button.button-del {
+          background: rgb(202, 60, 60);
+        }
+        .gropple-config button.button-add {
+          background: rgb(60, 200, 60);
+        }
+        .gropple-config .pure-form-message {
+          padding-top: .5em;
+          padding-bottom: 1.5em;
+        }
+        .error {
+          color: red;
+        }
+        .success {
+          color: green;
+        }
+
+        [x-cloak] { display: none !important; }
+
     </style>
 </head>
 <body style="margin:4; padding:4">
+
     {{ template "content" . }}
+    <footer>
+    Homepage: <a href="https://github.com/tardisx/gropple">https://github.com/tardisx/gropple</a>
+    </footer>
 </body>
     {{ template "js" . }}
 </html>
-{{ end }}
+{{ end }}

web/menu.tmpl

@@ -0,0 +1,15 @@
+    <div class="pure-menu pure-menu-horizontal" style="height: 2em;">
+        <a href="#" class="pure-menu-heading pure-menu-link">gropple</a>
+        <ul class="pure-menu-list">
+            <li class="pure-menu-item">
+                <a href="/" class="pure-menu-link">Home</a>
+            </li>
+            <li class="pure-menu-item">
+                <a href="/config" class="pure-menu-link">Config</a>
+            </li>
+            <li class="pure-menu-item">
+                <a href="https://github.com/tardisx/gropple" class="pure-menu-link">Github</a>
+            </li>
+        </ul>
+    </div>
+    

web/popup.html

@@ -1,8 +1,19 @@
 {{ define "content" }}
-    <div id="layout" class="pure-g pure-u-1" x-data="popup()"  x-init="fetch_data()">
+    <div id="layout" class="pure-g pure-u-1" x-data="popup()" x-init="fetch_data()">
         <h2>Download started</h2>
-        <p>Fetching <tt>{{ .Url }}</tt></p>
+        <p>Fetching <tt>{{ .dl.Url }}</tt></p>
         <table class="pure-table" >
+            <tr>
+                <th>profile</th>
+                <td>
+                    <select x-bind:disabled="profile_chosen" x-on:change="update_profile()" class="pure-input-1-2" x-model="profile_chosen">
+                        <option value="">choose a profile to start</option>
+                    {{ range $i := .config.DownloadProfiles }}
+                        <option>{{ $i.Name }}</option>
+                    {{ end }}
+                    </select>
+                </td>
+            </tr>
             <tr><th>current filename</th><td x-text="filename"></td></tr>
             <tr><th>state</th><td x-text="state"></td></tr>
             <tr><th>progress</th><td x-text="percent"></td></tr>
@@ -19,15 +30,36 @@
 {{ define "js" }}
 <script>
     function popup() {
+        history.replaceState(null, '', ['/fetch/{{ .dl.Id }}'])
         return { 
             eta: '', percent: 0.0, state: '??', filename: '', finished: false, log :'',
+            profile_chosen: null,
+            watch_profile() {
+                this.$watch('profile_chosen', value => this.profile_chosen(value))
+            },
+            update_profile(name) {
+                console.log('you chose name', this.profile_chosen);
+                let op = {
+                   method: 'POST',
+                   body: JSON.stringify({action: 'start', profile: this.profile_chosen}),
+                   headers: { 'Content-Type': 'application/json' }
+                };
+                fetch('/rest/fetch/{{ .dl.Id }}', op)
+                .then(response => response.json())
+                .then(info => {
+                    console.log(info)
+                })
+            },
             fetch_data() {
-                fetch('/fetch/info/{{ .Id }}')
+                fetch('/rest/fetch/{{ .dl.Id }}')
                 .then(response => response.json())
                 .then(info => {
                     this.eta = info.eta;
                     this.percent = info.percent + "%";
                     this.state = info.state;
+                    if (this.state != 'choose profile') {
+                        this.profile_chosen = true;
+                    }
                     this.finished = info.finished;
                     if (info.files && info.files.length > 0) {
                         this.filename = info.files[info.files.length - 1];