Implement download queue (default size 2) and cleanup old entries after a while

CHANGELOG.md

@@ -0,0 +1,49 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [Unreleased]
+
+## [v0.5.3] - 2021-11-21
+
+- Add config option to limit number of simultaneous downloads
+- Remove old download entries from the index after they are complete
+
+## [v0.5.2] - 2021-10-26
+
+- Provide link to re-display the popup window from the index
+- Visual improvements
+
+## [v0.5.1] - 2021-10-25
+
+- Add note about adblockers potentially blocking the popup
+- Make it possible to refresh the popup window without initiating a new download
+
+## [v0.5.0] - 2021-10-01
+
+- No more command line options, all configuration is now app-managed
+- Beautiful (ok, less ugly) new web interface
+- Multiple youtube-dl profiles, a profile can be chosen for each download
+- Bundled profiles include a standard video download and an mp3 download
+- Configuration via web interface, including download profile configuration
+
+## [v0.4.0] - 2021-09-26
+
+- Moved to semantic versioning
+- Automatic version check, prompts for upgrade in GUI
+- Fixed regex to properly match "merging" lines
+- Automatically refresh index page
+
+## [0.03] - 2021-09-24
+
+- Add option to change command (to use youtube-dlc or other forks) and command line arguments
+- Improve log display in popup
+- Improve documentation (slightly)
+
+## [0.02] - 2021-09-22
+
+- Fix #4 so that deleted files are removed from the results
+
+## [0.01] - 2021-09-22
+
+- Initial release

README.md

@@ -1,14 +1,14 @@
 # gropple
 
-A web service and bookmarklet to download videos with a single click.
+A frontend to youtube-dl (and forks) to download videos with a single click, straight from your web browser.
 
 ![Screencast](/screencast.gif)
 
 ## Pre-requisites
 
-* a passing familiarity with the command line
+* some familiarity with the command line
 * youtube-dl (plus any of its required dependencies, like ffmpeg)
-* golang compiler (if you'd like to build from source)
+* golang compiler (only if you'd like to build from source)
 
 ## Build
 
@@ -18,62 +18,70 @@ A web service and bookmarklet to download videos with a single click.
 
 Binaries are available at https://github.com/tardisx/gropple/releases
 
+Gropple will automatically check for available updates and prompt you to upgrade.
+
 ## Running
 
-    gropple -port 6283 -address http://hostname:6283 -path /downloads
+    ./gropple
 
-With no arguments, it will listen on port 6283 and use an address of 'http://localhost:6283'.
+There are no command line arguments. All configuration is done via the web
+interface. The address will be printed after startup:
 
-The address must be specified so that the bookmarklet can refer to the correct
-host if it is not running on your local machine. You may also need to specify
-a different address if you are running it behind a proxy server or similar.
+    2021/09/30 23:53:00 starting gropple v0.5.0 - https://github.com/tardisx/gropple
+    2021/09/30 23:53:00 go to http://localhost:6123 for details on installing the bookmarklet and to check status
 
 ## Using
 
-Bring up `http://localhost:6283` (or your chosen address) in your browser. You 
+Bring up `http://localhost:6283` (or your configured address) in your browser. You 
 should see a link to the bookmarklet at the top of the screen, and the list of
 downloads (currently empty).
 
 Drag the bookmarklet to your favourites bar, or otherwise bookmark it as you 
-see fit.
+see fit. Any kind of browser bookmark should work. The bookmarklet contains
+embedded javascript to pass the URL of whatever page you are currently on back
+to gropple.
 
-Whenever you are on a page with a video you would like to download, simply 
+So, whenever you are on a page with a video you would like to download just 
 click the bookmarklet.
 
-A popup window will appear, the download will start on the your gropple server 
-and the status will be shown in the window.
+A popup window will appear. Choose a download profile and the download will start.
+The status will be shown in the window, updating in real time.
 
 You may close this window at any time without stopping the download, the status 
 of all downloads is available on the index page.
 
-## Using an alternative downloader
+## Configuration
 
-The default downloader is youtube-dl. It is possible to use a different downloader 
-via the `-dl-cmd` command line option.
+Click the "config" link on the index page to configure gropple. The default options 
+are fine if you are running on your local machine. If you are running it remotely
+you will need to set the "server address" to ensure the bookmarklet has the correct
+URL in it.
 
-While `gropple` will use your `PATH` to find the executable, you may also want 
-to specify a full path instead.o
+### Configuring Downloaders
 
-So, for instance, to use `youtube-dlc` instead of `youtube-dl` and specify the 
-full path:
+Gropple's default configuration uses the original youtube-dl and has two profiles set
+up, one for downloading video, the other for downloading audio (mp3).
 
-`gropple -dl-cmd /home/username/bin/youtube-dlc`
+Note that gropple does not include any downloaders, you have to install them separately.
 
-Note that this is only the path to the executable. If you need to change the 
-command arguments, see below.
+If you would like to use a youtube-dl fork (like [yt-dlp](https://github.com/yt-dlp/yt-dlp))
+or change the options, you can do so on the right hand side. Create as many profiles as you 
+wish, whenever you start a download you can choose the appropriate profile.
 
-## Changing the youtube-dl arguments
+Note that the command arguments must each be specified separately - see the default configuration
+for an example.
 
-The default arguments passed to `youtube-dl` are:
+While gropple will use your `PATH` to find the executable, you can also specify a full path
+instead. Note that any tools that the downloader calls itself (for instance, ffmpeg) will 
+probably need to be available on your path.
 
-* `--newline` (needed to allow gropple to properly parse the output)
-* `--write-info-json` (optional, but provides information on the download in the corresponding .json file)
-* `-f` and `bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best` (choose the type of video `youtube-dl` will download)
+## Problems
 
-These are customisable on the command line for `gropple`. For example, to duplicate these default options, you would 
-do:
+Most download problems are probably diagnosable via the log - check in the popup window and scroll
+the log down to the bottom. The most common problem is that youtube-dl cannot be found, or its
+dependency (like ffmpeg) cannot be found on your path. 
 
-`gropple -dl-args '--newline' -dl-args '--write-info-json' -dl-args '-f' -dl-args 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best`
+For other problems, please file an issue on github.
 
 ## TODO
 

config/config.go

@@ -0,0 +1,243 @@
+package config
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"log"
+	"os"
+	"strings"
+
+	"gopkg.in/yaml.v2"
+)
+
+type Server struct {
+	Port                   int    `yaml:"port" json:"port"`
+	Address                string `yaml:"address" json:"address"`
+	DownloadPath           string `yaml:"download_path" json:"download_path"`
+	MaximumActiveDownloads int    `yaml:"maximum_active_downloads_per_domain" json:"maximum_active_downloads_per_domain"`
+}
+
+type DownloadProfile struct {
+	Name    string   `yaml:"name" json:"name"`
+	Command string   `yaml:"command" json:"command"`
+	Args    []string `yaml:"args" json:"args"`
+}
+
+type UI struct {
+	PopupWidth  int `yaml:"popup_width" json:"popup_width"`
+	PopupHeight int `yaml:"popup_height" json:"popup_height"`
+}
+
+type Config struct {
+	ConfigVersion    int               `yaml:"config_version" json:"config_version"`
+	Server           Server            `yaml:"server" json:"server"`
+	UI               UI                `yaml:"ui" json:"ui"`
+	DownloadProfiles []DownloadProfile `yaml:"profiles" json:"profiles"`
+}
+
+func TestConfig() *Config {
+	config := DefaultConfig()
+	config.DownloadProfiles = []DownloadProfile{{Name: "test profile", Command: "sleep", Args: []string{"5"}}}
+	return config
+}
+
+func DefaultConfig() *Config {
+	defaultConfig := Config{}
+	stdProfile := DownloadProfile{Name: "standard video", Command: "youtube-dl", Args: []string{
+		"--newline",
+		"--write-info-json",
+		"-f",
+		"bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best",
+	}}
+	mp3Profile := DownloadProfile{Name: "standard mp3", Command: "youtube-dl", Args: []string{
+		"--newline",
+		"--write-info-json",
+		"--extract-audio",
+		"--audio-format", "mp3",
+	}}
+
+	defaultConfig.DownloadProfiles = append(defaultConfig.DownloadProfiles, stdProfile)
+	defaultConfig.DownloadProfiles = append(defaultConfig.DownloadProfiles, mp3Profile)
+
+	defaultConfig.Server.Port = 6123
+	defaultConfig.Server.Address = "http://localhost:6123"
+	defaultConfig.Server.DownloadPath = "./"
+
+	defaultConfig.UI.PopupWidth = 500
+	defaultConfig.UI.PopupHeight = 500
+
+	defaultConfig.Server.MaximumActiveDownloads = 2
+
+	defaultConfig.ConfigVersion = 2
+
+	return &defaultConfig
+}
+
+func (c *Config) ProfileCalled(name string) *DownloadProfile {
+	for _, p := range c.DownloadProfiles {
+		if p.Name == name {
+			return &p
+		}
+	}
+	return nil
+}
+
+func (c *Config) UpdateFromJSON(j []byte) error {
+	newConfig := Config{}
+	err := json.Unmarshal(j, &newConfig)
+	if err != nil {
+		log.Printf("Unmarshal error in config: %v", err)
+		return err
+	}
+
+	// sanity checks
+	if newConfig.UI.PopupHeight < 100 || newConfig.UI.PopupHeight > 2000 {
+		return errors.New("invalid popup height - should be 100-2000")
+	}
+	if newConfig.UI.PopupWidth < 100 || newConfig.UI.PopupWidth > 2000 {
+		return errors.New("invalid popup width - should be 100-2000")
+	}
+
+	// check listen port
+	if newConfig.Server.Port < 1 || newConfig.Server.Port > 65535 {
+		return errors.New("invalid server listen port")
+	}
+
+	// check download path
+	fi, err := os.Stat(newConfig.Server.DownloadPath)
+	if os.IsNotExist(err) {
+		return fmt.Errorf("path '%s' does not exist", newConfig.Server.DownloadPath)
+	}
+	if !fi.IsDir() {
+		return fmt.Errorf("path '%s' is not a directory", newConfig.Server.DownloadPath)
+	}
+
+	if newConfig.Server.MaximumActiveDownloads < 0 {
+		return fmt.Errorf("maximum active downloads can not be < 0")
+	}
+
+	// check profile name uniqueness
+	for i, p1 := range newConfig.DownloadProfiles {
+		for j, p2 := range newConfig.DownloadProfiles {
+			if i != j && p1.Name == p2.Name {
+				return fmt.Errorf("duplicate download profile name '%s'", p1.Name)
+			}
+		}
+	}
+
+	// remove leading/trailing spaces from args and commands and check for emptiness
+	for i := range newConfig.DownloadProfiles {
+		newConfig.DownloadProfiles[i].Name = strings.TrimSpace(newConfig.DownloadProfiles[i].Name)
+
+		if newConfig.DownloadProfiles[i].Name == "" {
+			return errors.New("profile name cannot be empty")
+		}
+
+		newConfig.DownloadProfiles[i].Command = strings.TrimSpace(newConfig.DownloadProfiles[i].Command)
+		if newConfig.DownloadProfiles[i].Command == "" {
+			return fmt.Errorf("command in profile '%s' cannot be empty", newConfig.DownloadProfiles[i].Name)
+		}
+
+		// check the args
+		for j := range newConfig.DownloadProfiles[i].Args {
+			newConfig.DownloadProfiles[i].Args[j] = strings.TrimSpace(newConfig.DownloadProfiles[i].Args[j])
+			if newConfig.DownloadProfiles[i].Args[j] == "" {
+				return fmt.Errorf("argument %d of profile '%s' is empty", j+1, newConfig.DownloadProfiles[i].Name)
+			}
+		}
+
+	}
+
+	*c = newConfig
+	return nil
+}
+
+func configPath() string {
+	dir, err := os.UserConfigDir()
+	if err != nil {
+		log.Fatalf("cannot find a directory to store config: %v", err)
+	}
+	appDir := "gropple"
+
+	fullPath := dir + string(os.PathSeparator) + appDir
+	_, err = os.Stat(fullPath)
+
+	if os.IsNotExist(err) {
+		err := os.Mkdir(fullPath, 0777)
+		if err != nil {
+			log.Fatalf("Could not create config dir '%s': %v", fullPath, err)
+		}
+	}
+
+	fullFilename := fullPath + string(os.PathSeparator) + "config.yml"
+	return fullFilename
+}
+
+func ConfigFileExists() bool {
+	info, err := os.Stat(configPath())
+	if os.IsNotExist(err) {
+		return false
+	}
+	if err != nil {
+		log.Fatal(err)
+	}
+	if info.Size() == 0 {
+		log.Print("config file is 0 bytes?")
+		return false
+	}
+	return true
+}
+
+func LoadConfig() (*Config, error) {
+	path := configPath()
+	b, err := os.ReadFile(path)
+	if err != nil {
+		log.Printf("Could not read config '%s': %v", path, err)
+		return nil, err
+	}
+	c := Config{}
+	err = yaml.Unmarshal(b, &c)
+	if err != nil {
+		log.Printf("Could not parse YAML config '%s': %v", path, err)
+		return nil, err
+	}
+
+	// do migrations
+	configMigrated := false
+	if c.ConfigVersion == 1 {
+		c.Server.MaximumActiveDownloads = 2
+		c.ConfigVersion = 2
+		configMigrated = true
+		log.Print("migrated config from version 1 => 2")
+	}
+
+	if configMigrated {
+		log.Print("Writing new config after version migration")
+		c.WriteConfig()
+	}
+
+	return &c, nil
+}
+
+func (c *Config) WriteConfig() {
+	s, err := yaml.Marshal(c)
+	if err != nil {
+		panic(err)
+	}
+
+	path := configPath()
+	file, err := os.Create(
+		path,
+	)
+
+	if err != nil {
+		log.Fatalf("Could not open config file")
+	}
+	defer file.Close()
+
+	file.Write(s)
+	file.Close()
+
+	log.Printf("Wrote configuration out to %s", path)
+}

download/download.go

@@ -0,0 +1,233 @@
+package download
+
+import (
+	"fmt"
+	"io"
+	"os/exec"
+	"regexp"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/tardisx/gropple/config"
+)
+
+type Download struct {
+	Id              int                    `json:"id"`
+	Url             string                 `json:"url"`
+	PopupUrl        string                 `json:"popup_url"`
+	Pid             int                    `json:"pid"`
+	ExitCode        int                    `json:"exit_code"`
+	State           string                 `json:"state"`
+	DownloadProfile config.DownloadProfile `json:"download_profile"`
+	Finished        bool                   `json:"finished"`
+	FinishedTS      time.Time              `json:"finished_ts"`
+	Files           []string               `json:"files"`
+	Eta             string                 `json:"eta"`
+	Percent         float32                `json:"percent"`
+	Log             []string               `json:"log"`
+	Config          *config.Config
+}
+
+type Downloads []*Download
+
+// StartQueued starts any downloads that have been queued, we would not exceed
+// maxRunning. If maxRunning is 0, there is no limit.
+func (dls Downloads) StartQueued(maxRunning int) {
+	active := 0
+	queued := 0
+
+	for _, dl := range dls {
+		if dl.State == "downloading" {
+			active++
+		}
+		if dl.State == "queued" {
+			queued++
+		}
+	}
+
+	// there is room, so start one
+	if queued > 0 && (active < maxRunning || maxRunning == 0) {
+		for _, dl := range dls {
+			if dl.State == "queued" {
+				dl.State = "downloading"
+				go func() { dl.Begin() }()
+				return
+			}
+		}
+	}
+
+}
+
+// Cleanup removes old downloads from the list. Hardcoded to remove them one hour
+// completion.
+func (dls Downloads) Cleanup() Downloads {
+	newDLs := Downloads{}
+	for _, dl := range dls {
+		if dl.Finished && time.Since(dl.FinishedTS) > time.Duration(time.Hour) {
+			// do nothing
+		} else {
+			newDLs = append(newDLs, dl)
+		}
+	}
+	return newDLs
+}
+
+// Queue queues a download
+func (dl *Download) Queue() {
+	dl.State = "queued"
+}
+
+// Begin starts a download, by starting the command specified in the DownloadProfile.
+// It blocks until the download is complete.
+func (dl *Download) Begin() {
+	dl.State = "downloading"
+	cmdSlice := []string{}
+	cmdSlice = append(cmdSlice, dl.DownloadProfile.Args...)
+
+	// only add the url if it's not empty. This helps us with testing
+	if dl.Url != "" {
+		cmdSlice = append(cmdSlice, dl.Url)
+	}
+
+	cmd := exec.Command(dl.DownloadProfile.Command, cmdSlice...)
+	cmd.Dir = dl.Config.Server.DownloadPath
+
+	stdout, err := cmd.StdoutPipe()
+	if err != nil {
+		dl.State = "failed"
+		dl.Finished = true
+		dl.FinishedTS = time.Now()
+		dl.Log = append(dl.Log, fmt.Sprintf("error setting up stdout pipe: %v", err))
+		return
+	}
+
+	stderr, err := cmd.StderrPipe()
+	if err != nil {
+		dl.State = "failed"
+		dl.Finished = true
+		dl.FinishedTS = time.Now()
+		dl.Log = append(dl.Log, fmt.Sprintf("error setting up stderr pipe: %v", err))
+		return
+	}
+
+	err = cmd.Start()
+	if err != nil {
+		dl.State = "failed"
+		dl.Finished = true
+		dl.FinishedTS = time.Now()
+		dl.Log = append(dl.Log, fmt.Sprintf("error starting command '%s': %v", dl.DownloadProfile.Command, err))
+		return
+	}
+	dl.Pid = cmd.Process.Pid
+
+	var wg sync.WaitGroup
+
+	wg.Add(2)
+	go func() {
+		defer wg.Done()
+		dl.updateDownload(stdout)
+	}()
+
+	go func() {
+		defer wg.Done()
+		dl.updateDownload(stderr)
+	}()
+
+	wg.Wait()
+	cmd.Wait()
+
+	dl.State = "complete"
+	dl.Finished = true
+	dl.FinishedTS = time.Now()
+	dl.ExitCode = cmd.ProcessState.ExitCode()
+
+	if dl.ExitCode != 0 {
+		dl.State = "failed"
+	}
+
+}
+
+func (dl *Download) updateDownload(r io.Reader) {
+	// XXX not sure if we might get a partial line?
+	buf := make([]byte, 1024)
+	for {
+		n, err := r.Read(buf)
+		if n > 0 {
+			s := string(buf[:n])
+			lines := strings.Split(s, "\n")
+
+			for _, l := range lines {
+
+				if l == "" {
+					continue
+				}
+
+				// append the raw log
+				dl.Log = append(dl.Log, l)
+
+				// look for the percent and eta and other metadata
+				dl.updateMetadata(l)
+			}
+		}
+		if err != nil {
+			break
+		}
+	}
+}
+
+func (dl *Download) updateMetadata(s string) {
+
+	// [download]  49.7% of ~15.72MiB at  5.83MiB/s ETA 00:07
+	etaRE := regexp.MustCompile(`download.+ETA +(\d\d:\d\d(?::\d\d)?)$`)
+	matches := etaRE.FindStringSubmatch(s)
+	if len(matches) == 2 {
+		dl.Eta = matches[1]
+		dl.State = "downloading"
+
+	}
+
+	percentRE := regexp.MustCompile(`download.+?([\d\.]+)%`)
+	matches = percentRE.FindStringSubmatch(s)
+	if len(matches) == 2 {
+		p, err := strconv.ParseFloat(matches[1], 32)
+		if err == nil {
+			dl.Percent = float32(p)
+		} else {
+			panic(err)
+		}
+	}
+
+	// This appears once per destination file
+	// [download] Destination: Filename with spaces and other punctuation here be careful!.mp4
+	filename := regexp.MustCompile(`download.+?Destination: (.+)$`)
+	matches = filename.FindStringSubmatch(s)
+	if len(matches) == 2 {
+		dl.Files = append(dl.Files, matches[1])
+	}
+
+	// This means a file has been "created" by merging others
+	// [ffmpeg] Merging formats into "Toto - Africa (Official HD Video)-FTQbiNvZqaY.mp4"
+	mergedFilename := regexp.MustCompile(`Merging formats into "(.+)"$`)
+	matches = mergedFilename.FindStringSubmatch(s)
+	if len(matches) == 2 {
+		dl.Files = append(dl.Files, matches[1])
+	}
+
+	// This means a file has been deleted
+	// Gross - this time it's unquoted and has trailing guff
+	// Deleting original file Toto - Africa (Official HD Video)-FTQbiNvZqaY.f137.mp4 (pass -k to keep)
+	// This is very fragile
+	deletedFile := regexp.MustCompile(`Deleting original file (.+) \(pass -k to keep\)$`)
+	matches = deletedFile.FindStringSubmatch(s)
+	if len(matches) == 2 {
+		// find the index
+		for i, f := range dl.Files {
+			if f == matches[1] {
+				dl.Files = append(dl.Files[:i], dl.Files[i+1:]...)
+				break
+			}
+		}
+	}
+}

download/download_test.go

@@ -0,0 +1,99 @@
+package download
+
+import (
+	"testing"
+
+	"github.com/tardisx/gropple/config"
+)
+
+func TestUpdateMetadata(t *testing.T) {
+	newD := Download{}
+
+	// first time we spot a filename
+	newD.updateMetadata("[download] Destination: Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.f401.mp4")
+	if len(newD.Files) != 1 || newD.Files[0] != "Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.f401.mp4" {
+		t.Fatalf("incorrect Files:%v", newD.Files)
+	}
+
+	// eta's might be xx:xx:xx or xx:xx
+	newD.updateMetadata("[download]   0.0% of 504.09MiB at 135.71KiB/s ETA 01:03:36")
+	if newD.Eta != "01:03:36" {
+		t.Fatalf("bad long eta in dl\n%v", newD)
+	}
+	newD.updateMetadata("[download]   0.0% of 504.09MiB at 397.98KiB/s ETA 21:38")
+	if newD.Eta != "21:38" {
+		t.Fatalf("bad short eta in dl\n%v", newD)
+	}
+
+	// added a new file, now we are tracking two
+	newD.updateMetadata("[download] Destination: Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.f140.m4a")
+	if len(newD.Files) != 2 || newD.Files[1] != "Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.f140.m4a" {
+		t.Fatalf("incorrect Files:%v", newD.Files)
+	}
+
+	// merging
+	newD.updateMetadata("[ffmpeg] Merging formats into \"Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.mp4\"")
+	if len(newD.Files) != 3 || newD.Files[2] != "Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.mp4" {
+		t.Fatalf("did not find merged filename")
+		t.Fatalf("%v", newD.Files)
+	}
+
+	// deletes
+	// TODO. Not sure why I don't always see the "Deleting original file" messages after merge -
+	// maybe a youtube-dl fork thing?
+
+}
+
+// [youtube] wi7Agv1M6PY: Downloading webpage
+// [info] Writing video description metadata as JSON to: Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.info.json
+// [download] Destination: Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.f401.mp4
+// [download]   0.0% of 504.09MiB at 135.71KiB/s ETA 01:03:36
+// [download]   0.0% of 504.09MiB at 397.98KiB/s ETA 21:38
+// [download]   0.0% of 504.09MiB at 918.97KiB/s ETA 09:22
+// [download]   0.0% of 504.09MiB at  1.90MiB/s ETA 04:25
+// ..
+// [download]  99.6% of 504.09MiB at  8.91MiB/s ETA 00:00
+// [download] 100.0% of 504.09MiB at  9.54MiB/s ETA 00:00
+// [download] 100% of 504.09MiB in 01:00
+// [download] Destination: Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.f140.m4a
+// [download]   0.0% of 4.64MiB at 155.26KiB/s ETA 00:30
+// [download]   0.1% of 4.64MiB at 457.64KiB/s ETA 00:10
+// [download]   0.1% of 4.64MiB at  1.03MiB/s ETA 00:04
+// ..
+// [download]  86.2% of 4.64MiB at 10.09MiB/s ETA 00:00
+// [download] 100.0% of 4.64MiB at 10.12MiB/s ETA 00:00
+// [download] 100% of 4.64MiB in 00:00
+// [ffmpeg] Merging formats into "Halo Infinite Flight 4K Gameplay-wi7Agv1M6PY.mp4"
+
+// This test is a bit broken, because StartQueued immediately starts the queued
+// download, it
+func TestQueue(t *testing.T) {
+	conf := config.TestConfig()
+
+	new1 := Download{Id: 1, State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
+	new2 := Download{Id: 2, State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
+	new3 := Download{Id: 3, State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
+	new4 := Download{Id: 4, State: "queued", DownloadProfile: conf.DownloadProfiles[0], Config: conf}
+
+	dls := Downloads{&new1, &new2, &new3, &new4}
+	dls.StartQueued(1)
+	if dls[0].State == "queued" {
+		t.Error("#1 was not started")
+	}
+	if dls[1].State != "queued" {
+		t.Error("#2 is not queued")
+	}
+
+	// this should start no more, as one is still going
+	dls.StartQueued(1)
+	if dls[1].State != "queued" {
+		t.Error("#2 was started when it should not be")
+	}
+
+	dls.StartQueued(2)
+	if dls[1].State == "queued" {
+		t.Error("#2 was not started but it should be")
+
+	}
+
+}

go.mod

@@ -5,4 +5,5 @@ go 1.16
 require (
 	github.com/gorilla/mux v1.8.0
 	golang.org/x/mod v0.5.1
+	gopkg.in/yaml.v2 v2.4.0
 )

go.sum

@@ -13,3 +13,7 @@ golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
+gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=

main.go

@@ -3,94 +3,72 @@ package main
 import (
 	"embed"
 	"encoding/json"
-	"flag"
 	"fmt"
 	"html/template"
 	"io"
 	"log"
 	"net/http"
-	"os/exec"
-	"regexp"
 	"strings"
-	"sync"
 	"time"
 
 	"strconv"
 
 	"github.com/gorilla/mux"
+	"github.com/tardisx/gropple/config"
+	"github.com/tardisx/gropple/download"
 	"github.com/tardisx/gropple/version"
 )
 
-type download struct {
-	Id       int      `json:"id"`
-	Url      string   `json:"url"`
-	Pid      int      `json:"pid"`
-	ExitCode int      `json:"exit_code"`
-	State    string   `json:"state"`
-	Finished bool     `json:"finished"`
-	Files    []string `json:"files"`
-	Eta      string   `json:"eta"`
-	Percent  float32  `json:"percent"`
-	Log      []string `json:"log"`
-}
-
-var downloads []*download
+var downloads download.Downloads
 var downloadId = 0
-var downloadPath = "./"
+var conf *config.Config
 
-var address string
-
-var dlCmd = "youtube-dl"
-
-type args []string
-
-var dlArgs = args{}
-var defaultArgs = args{
-	"--write-info-json",
-	"-f", "bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best",
-	"--newline",
-}
-
-var versionInfo = version.Info{CurrentVersion: "v0.4.0"}
+var versionInfo = version.Info{CurrentVersion: "v0.5.3"}
 
 //go:embed web
 var webFS embed.FS
 
-func (i *args) Set(value string) error {
-	*i = append(*i, strings.TrimSpace(value))
-	return nil
+type successResponse struct {
+	Success bool   `json:"success"`
+	Message string `json:"message"`
 }
 
-func (i *args) String() string {
-	return strings.Join(*i, ",")
+type errorResponse struct {
+	Success bool   `json:"success"`
+	Error   string `json:"error"`
 }
 
 func main() {
-	var port int
-	flag.IntVar(&port, "port", 6283, "port to listen on")
-	flag.StringVar(&address, "address", "http://localhost:6283", "address for the service")
-	flag.StringVar(&downloadPath, "path", "", "path for downloaded files - defaults to current directory")
-	flag.StringVar(&dlCmd, "dl-cmd", "youtube-dl", "downloader to use")
-	flag.Var(&dlArgs, "dl-args", "arguments to the downloader")
-
-	flag.Parse()
-
-	if len(dlArgs) == 0 {
-		dlArgs = defaultArgs
+	if !config.ConfigFileExists() {
+		log.Print("No config file - creating default config")
+		conf = config.DefaultConfig()
+		conf.WriteConfig()
+	} else {
+		loadedConfig, err := config.LoadConfig()
+		if err != nil {
+			log.Fatal(err)
+		}
+		conf = loadedConfig
 	}
 
 	r := mux.NewRouter()
-	r.HandleFunc("/", HomeHandler)
-	r.HandleFunc("/fetch", FetchHandler)
-	r.HandleFunc("/fetch/info", FetchInfoHandler)
-	r.HandleFunc("/fetch/info/{id}", FetchInfoOneHandler)
-	r.HandleFunc("/version", VersionHandler)
+	r.HandleFunc("/", homeHandler)
+	r.HandleFunc("/config", configHandler)
+	r.HandleFunc("/fetch", fetchHandler)
+	r.HandleFunc("/fetch/{id}", fetchHandler)
+
+	// info for the list
+	r.HandleFunc("/rest/fetch", fetchInfoRESTHandler)
+	// info for one, including update
+	r.HandleFunc("/rest/fetch/{id}", fetchInfoOneRESTHandler)
+	r.HandleFunc("/rest/version", versionRESTHandler)
+	r.HandleFunc("/rest/config", configRESTHandler)
 
 	http.Handle("/", r)
 
 	srv := &http.Server{
 		Handler: r,
-		Addr:    fmt.Sprintf(":%d", port),
+		Addr:    fmt.Sprintf(":%d", conf.Server.Port),
 		// Good practice: enforce timeouts for servers you create!
 		WriteTimeout: 5 * time.Second,
 		ReadTimeout:  5 * time.Second,
@@ -104,12 +82,23 @@ func main() {
 		}
 	}()
 
+	// start downloading queued downloads when slots available, and clean up
+	// old entries
+	go func() {
+		for {
+			downloads.StartQueued(conf.Server.MaximumActiveDownloads)
+			downloads = downloads.Cleanup()
+			time.Sleep(time.Second)
+		}
+	}()
+
 	log.Printf("starting gropple %s - https://github.com/tardisx/gropple", versionInfo.CurrentVersion)
-	log.Printf("go to %s for details on installing the bookmarklet and to check status", address)
+	log.Printf("go to %s for details on installing the bookmarklet and to check status", conf.Server.Address)
 	log.Fatal(srv.ListenAndServe())
 }
 
-func VersionHandler(w http.ResponseWriter, r *http.Request) {
+// versionRESTHandler returns the version information, if we have up-to-date info from github
+func versionRESTHandler(w http.ResponseWriter, r *http.Request) {
 	if versionInfo.GithubVersionFetched {
 		b, _ := json.Marshal(versionInfo)
 		w.Write(b)
@@ -118,34 +107,76 @@ func VersionHandler(w http.ResponseWriter, r *http.Request) {
 	}
 }
 
-func HomeHandler(w http.ResponseWriter, r *http.Request) {
+// homeHandler returns the main index page
+func homeHandler(w http.ResponseWriter, r *http.Request) {
 	w.WriteHeader(http.StatusOK)
 
-	bookmarkletURL := fmt.Sprintf("javascript:(function(f,s,n,o){window.open(f+encodeURIComponent(s),n,o)}('%s/fetch?url=',window.location,'yourform','width=500,height=500'));", address)
+	bookmarkletURL := fmt.Sprintf("javascript:(function(f,s,n,o){window.open(f+encodeURIComponent(s),n,o)}('%s/fetch?url=',window.location,'yourform','width=%d,height=%d'));", conf.Server.Address, conf.UI.PopupWidth, conf.UI.PopupHeight)
 
-	t, err := template.ParseFS(webFS, "web/layout.tmpl", "web/index.html")
+	t, err := template.ParseFS(webFS, "web/layout.tmpl", "web/menu.tmpl", "web/index.html")
 	if err != nil {
 		panic(err)
 	}
 
 	type Info struct {
-		Downloads      []*download
+		Downloads      []*download.Download
 		BookmarkletURL template.URL
+		Config         *config.Config
 	}
 
 	info := Info{
 		Downloads:      downloads,
 		BookmarkletURL: template.URL(bookmarkletURL),
+		Config:         conf,
 	}
 
 	err = t.ExecuteTemplate(w, "layout", info)
 	if err != nil {
 		panic(err)
 	}
-
 }
 
-func FetchInfoOneHandler(w http.ResponseWriter, r *http.Request) {
+// configHandler returns the configuration page
+func configHandler(w http.ResponseWriter, r *http.Request) {
+	w.WriteHeader(http.StatusOK)
+
+	t, err := template.ParseFS(webFS, "web/layout.tmpl", "web/menu.tmpl", "web/config.html")
+	if err != nil {
+		panic(err)
+	}
+
+	err = t.ExecuteTemplate(w, "layout", nil)
+	if err != nil {
+		panic(err)
+	}
+}
+
+// configRESTHandler handles both reading and writing of the configuration
+func configRESTHandler(w http.ResponseWriter, r *http.Request) {
+
+	if r.Method == "POST" {
+		log.Printf("Updating config")
+		b, err := io.ReadAll(r.Body)
+		if err != nil {
+			panic(err)
+		}
+		err = conf.UpdateFromJSON(b)
+
+		if err != nil {
+			errorRes := errorResponse{Success: false, Error: err.Error()}
+			errorResB, _ := json.Marshal(errorRes)
+			w.WriteHeader(400)
+			w.Write(errorResB)
+			return
+		}
+		conf.WriteConfig()
+	}
+	b, _ := json.Marshal(conf)
+	w.Write(b)
+}
+
+//
+func fetchInfoOneRESTHandler(w http.ResponseWriter, r *http.Request) {
 	vars := mux.Vars(r)
 	idString := vars["id"]
 	if idString != "" {
@@ -155,24 +186,99 @@ func FetchInfoOneHandler(w http.ResponseWriter, r *http.Request) {
 			return
 		}
 
+		// find the download
+		var thisDownload *download.Download
 		for _, dl := range downloads {
 			if dl.Id == id {
-				b, _ := json.Marshal(dl)
-				w.Write(b)
+				thisDownload = dl
+			}
+		}
+		if thisDownload == nil {
+			http.NotFound(w, r)
+			return
+		}
+
+		if r.Method == "POST" {
+
+			type updateRequest struct {
+				Action  string `json:"action"`
+				Profile string `json:"profile"`
+			}
+
+			thisReq := updateRequest{}
+
+			b, err := io.ReadAll(r.Body)
+			if err != nil {
+				panic(err)
+			}
+
+			err = json.Unmarshal(b, &thisReq)
+			if err != nil {
+				errorRes := errorResponse{Success: false, Error: err.Error()}
+				errorResB, _ := json.Marshal(errorRes)
+				w.WriteHeader(400)
+				w.Write(errorResB)
+				return
+			}
+
+			if thisReq.Action == "start" {
+				// find the profile they asked for
+				profile := conf.ProfileCalled(thisReq.Profile)
+				if profile == nil {
+					panic("bad profile name?")
+				}
+				// set the profile
+				thisDownload.DownloadProfile = *profile
+
+				thisDownload.Queue()
+				succRes := successResponse{Success: true, Message: "download started"}
+				succResB, _ := json.Marshal(succRes)
+				w.Write(succResB)
 				return
 			}
 		}
+
+		// just a get, return the object
+		b, _ := json.Marshal(thisDownload)
+		w.Write(b)
+		return
 	} else {
 		http.NotFound(w, r)
 	}
 }
 
-func FetchInfoHandler(w http.ResponseWriter, r *http.Request) {
+func fetchInfoRESTHandler(w http.ResponseWriter, r *http.Request) {
 	b, _ := json.Marshal(downloads)
 	w.Write(b)
 }
 
-func FetchHandler(w http.ResponseWriter, r *http.Request) {
+func fetchHandler(w http.ResponseWriter, r *http.Request) {
+
+	// if they refreshed the popup, just load the existing object, don't
+	// create a new one
+	vars := mux.Vars(r)
+	idString := vars["id"]
+
+	idInt, err := strconv.ParseInt(idString, 10, 32)
+	if err == nil && idInt > 0 {
+		for _, dl := range downloads {
+			if dl.Id == int(idInt) {
+				t, err := template.ParseFS(webFS, "web/layout.tmpl", "web/popup.html")
+				if err != nil {
+					panic(err)
+				}
+
+				templateData := map[string]interface{}{"dl": dl, "config": conf}
+
+				err = t.ExecuteTemplate(w, "layout", templateData)
+				if err != nil {
+					panic(err)
+				}
+				return
+			}
+		}
+
+	}
 
 	query := r.URL.Query()
 	url, present := query["url"]
@@ -184,7 +290,7 @@ func FetchHandler(w http.ResponseWriter, r *http.Request) {
 	} else {
 
 		// check the URL for a sudden but inevitable betrayal
-		if strings.Contains(url[0], address) {
+		if strings.Contains(url[0], conf.Server.Address) {
 			w.WriteHeader(400)
 			fmt.Fprint(w, "you musn't gropple your gropple :-)")
 			return
@@ -193,10 +299,13 @@ func FetchHandler(w http.ResponseWriter, r *http.Request) {
 		// create the record
 		// XXX should be atomic!
 		downloadId++
-		newDownload := download{
+		newDownload := download.Download{
+			Config: conf,
+
 			Id:       downloadId,
 			Url:      url[0],
-			State:    "starting",
+			PopupUrl: fmt.Sprintf("/fetch/%d", downloadId),
+			State:    "choose profile",
 			Finished: false,
 			Eta:      "?",
 			Percent:  0.0,
@@ -207,160 +316,20 @@ func FetchHandler(w http.ResponseWriter, r *http.Request) {
 
 		newDownload.Log = append(newDownload.Log, "start of log...")
 
-		go func() {
-			queue(&newDownload)
-		}()
+		// go func() {
+		// 	newDownload.Begin()
+		// }()
 
 		t, err := template.ParseFS(webFS, "web/layout.tmpl", "web/popup.html")
 		if err != nil {
 			panic(err)
 		}
-		err = t.ExecuteTemplate(w, "layout", newDownload)
+
+		templateData := map[string]interface{}{"dl": newDownload, "config": conf}
+
+		err = t.ExecuteTemplate(w, "layout", templateData)
 		if err != nil {
 			panic(err)
 		}
 	}
 }
-
-func queue(dl *download) {
-	cmdSlice := []string{}
-	cmdSlice = append(cmdSlice, dlArgs...)
-	cmdSlice = append(cmdSlice, dl.Url)
-
-	cmd := exec.Command(dlCmd, cmdSlice...)
-	cmd.Dir = downloadPath
-
-	stdout, err := cmd.StdoutPipe()
-	if err != nil {
-		dl.State = "failed"
-		dl.Finished = true
-		dl.Log = append(dl.Log, fmt.Sprintf("error setting up stdout pipe: %v", err))
-		return
-	}
-
-	stderr, err := cmd.StderrPipe()
-	if err != nil {
-		dl.State = "failed"
-		dl.Finished = true
-		dl.Log = append(dl.Log, fmt.Sprintf("error setting up stderr pipe: %v", err))
-		return
-	}
-
-	err = cmd.Start()
-	if err != nil {
-		dl.State = "failed"
-		dl.Finished = true
-		dl.Log = append(dl.Log, fmt.Sprintf("error starting youtube-dl: %v", err))
-		return
-	}
-	dl.Pid = cmd.Process.Pid
-
-	var wg sync.WaitGroup
-
-	wg.Add(2)
-	go func() {
-		defer wg.Done()
-		updateDownload(stdout, dl)
-	}()
-
-	go func() {
-		defer wg.Done()
-		updateDownload(stderr, dl)
-	}()
-
-	wg.Wait()
-	cmd.Wait()
-
-	dl.State = "complete"
-	dl.Finished = true
-	dl.ExitCode = cmd.ProcessState.ExitCode()
-
-	if dl.ExitCode != 0 {
-		dl.State = "failed"
-	}
-
-}
-
-func updateDownload(r io.Reader, dl *download) {
-	// XXX not sure if we might get a partial line?
-	buf := make([]byte, 1024)
-	for {
-		n, err := r.Read(buf)
-		if n > 0 {
-			s := string(buf[:n])
-			lines := strings.Split(s, "\n")
-
-			for _, l := range lines {
-
-				if l == "" {
-					continue
-				}
-
-				// append the raw log
-				dl.Log = append(dl.Log, l)
-
-				// look for the percent and eta and other metadata
-				updateMetadata(dl, l)
-			}
-		}
-		if err != nil {
-			break
-		}
-	}
-}
-
-func updateMetadata(dl *download, s string) {
-
-	// [download]  49.7% of ~15.72MiB at  5.83MiB/s ETA 00:07
-	etaRE := regexp.MustCompile(`download.+ETA +(\d\d:\d\d)`)
-	matches := etaRE.FindStringSubmatch(s)
-	if len(matches) == 2 {
-		dl.Eta = matches[1]
-		dl.State = "downloading"
-
-	}
-
-	percentRE := regexp.MustCompile(`download.+?([\d\.]+)%`)
-	matches = percentRE.FindStringSubmatch(s)
-	if len(matches) == 2 {
-		p, err := strconv.ParseFloat(matches[1], 32)
-		if err == nil {
-			dl.Percent = float32(p)
-		} else {
-			panic(err)
-		}
-	}
-
-	// This appears once per destination file
-	// [download] Destination: Filename with spaces and other punctuation here be careful!.mp4
-	filename := regexp.MustCompile(`download.+?Destination: (.+)$`)
-	matches = filename.FindStringSubmatch(s)
-	if len(matches) == 2 {
-		dl.Files = append(dl.Files, matches[1])
-	}
-
-	// This means a file has been "created" by merging others
-	// [ffmpeg] Merging formats into "Toto - Africa (Official HD Video)-FTQbiNvZqaY.mp4"
-	mergedFilename := regexp.MustCompile(`Merging formats into "(.+)"$`)
-	matches = mergedFilename.FindStringSubmatch(s)
-	if len(matches) == 2 {
-		dl.Files = append(dl.Files, matches[1])
-	}
-
-	// This means a file has been deleted
-	// Gross - this time it's unquoted and has trailing guff
-	// Deleting original file Toto - Africa (Official HD Video)-FTQbiNvZqaY.f137.mp4 (pass -k to keep)
-	// This is very fragile
-	deletedFile := regexp.MustCompile(`Deleting original file (.+) \(pass -k to keep\)$`)
-	matches = deletedFile.FindStringSubmatch(s)
-	if len(matches) == 2 {
-		// find the index
-		for i, f := range dl.Files {
-			if f == matches[1] {
-				dl.Files = append(dl.Files[:i], dl.Files[i+1:]...)
-				break
-			}
-		}
-	}
-
-}

version/version.go

@@ -24,7 +24,8 @@ func (i *Info) UpdateGitHubVersion() error {
 	versionUrl := "https://api.github.com/repos/tardisx/gropple/releases"
 	resp, err := http.Get(versionUrl)
 	if err != nil {
-		log.Fatal("Error getting response. ", err)
+		log.Printf("Error getting response: %v", err)
+		return err
 	}
 	defer resp.Body.Close()
 
@@ -65,11 +66,11 @@ func (i *Info) canUpgrade() bool {
 	log.Printf("We are %s, github is %s", i.CurrentVersion, i.GithubVersion)
 
 	if !semver.IsValid(i.CurrentVersion) {
-		log.Fatalf("current version %s is invalid", i.CurrentVersion)
+		log.Printf("current version %s is invalid", i.CurrentVersion)
 	}
 
 	if !semver.IsValid(i.GithubVersion) {
-		log.Fatalf("github version %s is invalid", i.GithubVersion)
+		log.Printf("github version %s is invalid", i.GithubVersion)
 	}
 
 	if semver.Compare(i.CurrentVersion, i.GithubVersion) == -1 {

web/config.html

@@ -0,0 +1,164 @@
+{{ define "content" }}
+
+{{ template "menu.tmpl" . }}
+
+<div x-data="config()" x-init="fetch_config();">
+
+    <p class="error"  x-show="error_message"  x-transition.duration.500ms x-text="error_message"></p>
+    <p class="success" x-show="success_message" x-transition.duration.500ms x-text="success_message"></p>
+
+    <p>Note: changes are not saved until the "Save Config" button is pressed at the bottom of the page.</p>
+
+    <div class="pure-g">
+
+        <div class="pure-u-md-1-2 pure-u-1">
+   
+            <form class="pure-form pure-form-stacked gropple-config">
+                <fieldset>
+
+                    <legend>Server</legend>
+
+                    <label for="config-server-port">Listen Port</label>
+                    <input type="text" id="config-server-port" placeholder="port number" x-model.number="config.server.port" />
+                    <span class="pure-form-message">The port the web server will listen on.</span>
+
+                    <label for="config-server-address">Server address (URL)</label>
+                    <input type="text" id="config-server-address" class="input-long" placeholder="server address" x-model="config.server.address" />
+                    <span class="pure-form-message">
+                        The address the service will be available on. Generally it will be http://hostname:port where
+                        hostname is the host the server is running on, and port is the port you set above.
+                    </span>
+
+                    <label for="config-server-downloadpath">Download path</label>
+                    <input type="text" id="config-server-downloadpath" placeholder="path" class="input-long" x-model="config.server.download_path" />
+                    <span class="pure-form-message">The path on the server to download files to.</span>
+
+                    <label for="config-server-max-downloads">Maximum active downloads</label>
+                    <input type="text" id="config-server-max-downloads" placeholder="2" class="input-long" x-model.number="config.server.maximum_active_downloads_per_domain" />
+                    <span class="pure-form-message">How many downloads can be simultaneously active. Use '0' for no limit.</span>
+
+                    <legend>UI</legend>
+
+                    <p>Note that changes to the popup dimensions will require you to recreate your bookmarklet.</p>
+
+                    <label for="config-ui-popupwidth">Popup Width</label>
+                    <input type="text" id="config-ui-popupwidth" placeholder="width in pixels" x-model.number="config.ui.popup_width" />
+                    <span class="pure-form-message">The width of popup windows in pixels.</span>
+
+                    <label for="config-ui-popupheight">Popup Height</label>
+                    <input type="text" id="config-ui-popupheight" placeholder="height in pixels" x-model.number="config.ui.popup_height" />
+                    <span class="pure-form-message">The height of popup windows in pixels.</span>
+
+                </fieldset>
+            </form>
+
+        </div>
+        <div class="pure-u-md-1-2 pure-u-1">
+            <form class="pure-form gropple-config">
+                <fieldset>
+
+                    <legend>Download Profiles</legend>
+
+                    <p>Gropple supports multiple download profiles. Each profile specifies a different youtube-dl 
+                        compatible command, and arguments. When starting a download, you may choose which profile 
+                        to use. The URL will be appended to the argument list at the end.
+                    </p>
+
+                    <hr>
+
+                    <template x-for="(profile, i) in config.profiles">
+                        <div>
+                            <label x-bind:for="'config-profiles-'+i+'-name'">Name of profile <span x-text="i+1"></span>
+                            </label>                                
+                            
+                            <input type="text" x-bind:id="'config-profiles-'+i+'-name'" class="input-long" placeholder="name" x-model="profile.name" />
+                            <button class="pure-button button-del" href="#" @click.prevent="config.profiles.splice(i, 1);;">delete profile</button>
+
+                            <span class="pure-form-message">The name of this profile. For your information only.</span>
+
+                            <label x-bind:for="'config-profiles-'+i+'-command'">Command to run</label>
+                            <input type="text" x-bind:id="'config-profiles-'+i+'-command'" class="input-long" placeholder="name" x-model="profile.command" />
+                            <span class="pure-form-message">Which command to run. Your path will be searched, or you can specify the full path here.</span>
+
+
+                            <label>Arguments</label>
+
+                            <template x-for="(arg, j) in profile.args">
+                                <div>
+                                    <input type="text" x-bind:id="'config-profiles-'+i+'-arg-'+j" placeholder="arg" x-model="profile.args[j]" />
+                                    <button class="pure-button button-del" href="#" @click.prevent="profile.args.splice(j, 1);;">delete arg</button>
+                                </div>
+                            </template>
+
+                            <button class="pure-button button-add" href="#" @click.prevent="profile.args.push('');">add arg</button>
+
+                            <hr>
+
+                        </div>
+                    </template>
+
+                    <button class="pure-button button-add" href="#" @click.prevent="config.profiles.push({name: 'new profile', command: 'youtube-dl', args: []});">add profile</button>
+
+                </fieldset>
+            </form>
+        </div>
+    </div>
+    <div class="pure-g">
+        <div class="pure-u-1">
+            <button class="pure-button pure-button-primary" @click="save_config();" href="#">Save Config</button>
+        </div>
+    </div>
+
+
+</div>
+
+{{ end }}
+
+{{ define "js" }}
+<script>
+    function config() {
+        return { 
+            config: { server : {}, ui : {}, profiles: [] }, 
+            error_message: '',
+            success_message: '',
+
+            fetch_config() {
+                fetch('/rest/config')
+                .then(response => response.json())
+                .then(config => {
+                    this.config = config;
+                })
+                .catch(error => {
+                    console.log('failed to fetch config', error);
+                });
+            },
+            save_config() {
+                let op = {
+                   method: 'POST',
+                   body: JSON.stringify(this.config),
+                   headers: { 'Content-Type': 'application/json' }
+                };
+                fetch('/rest/config', op)
+                .then(response => {
+                    return response.json();
+                })
+                .then(response => {
+                    if (response.error) {
+                        this.error_message = response.error;
+                        this.success_message = '';
+                        document.body.scrollTop = document.documentElement.scrollTop = 0;
+                    } else {
+                        this.error_message = '';
+                        this.success_message = 'configuration saved';
+                        document.body.scrollTop = document.documentElement.scrollTop = 0;
+                        this.config = response;
+                    }
+                })
+                .catch(error => {
+                    console.log('exception' ,error);
+                });
+            }
+        }
+    }
+</script>
+{{ end }}

web/index.html

@@ -1,11 +1,10 @@
 {{ define "content" }}
 
+{{ template "menu.tmpl" . }}
 
 <div x-data="index()" x-init="fetch_data(); fetch_version()">
 
-    <h2>gropple</h2>
-
-    <p x-show="version && version.upgrade_available">
+    <p x-cloak x-show="version && version.upgrade_available">
         <a href="https://github.com/tardisx/gropple/releases">Upgrade is available</a> - 
         you have 
         <span x-text="version.current_version"></span> and 
@@ -17,12 +16,15 @@
             Drag this bookmarklet: <a href="{{ .BookmarkletURL }}">Gropple</a> to your bookmark bar, and click it
             on any page you want to grab the video from.
         </p>
+	<p>
+	    Please note that some adblockers may prevent the bookmarklet from opening the popup window.
+	</p>
     </div>
 
     <table class="pure-table">
         <thead>
             <tr>
-                <th>id</th><th>filename</th><th>url</th><th>state</th><th>percent</th><th>eta</th><th>finished</th>
+		    <th>id</th><th>filename</th><th>url</th><th>show</th><th>state</th><th>percent</th><th>eta</th><th>finished</th>
             </tr>
         </thead>
         <tbody>
@@ -30,11 +32,12 @@
                 <tr>
                     <td x-text="item.id"></td>
                     <td x-text="item.files"></td>
-                    <td><a x-bind:href="item.url">link</a></td>
-                    <td x-text="item.state"></td>
+                    <td><a class="int-link" x-bind:href="item.url">&#x2197;</a></td>
+                    <td><a class="int-link" @click="show_popup(item)" href="#">&#x1F4C4;</a></td>
+                    <td :class="'state-'+item.state" x-text="item.state"></td>
                     <td x-text="item.percent"></td>
                     <td x-text="item.eta"></td>
-                    <td x-text="item.finished"></td>
+                    <td x-text="item.finished ? '&#x2714;' : '-'"></td>
                 </tr>
     
             </template>
@@ -53,7 +56,7 @@
         return { 
             items: [], version: {},
             fetch_version() {
-                fetch('/version')
+                fetch('/rest/version')
                 .then(response => response.json())
                 .then(info => {
                     this.version = info;
@@ -65,7 +68,7 @@
                 });
             },
             fetch_data() {
-                fetch('/fetch/info')
+                fetch('/rest/fetch')
                 .then(response => response.json())
                 .then(info => {
                     // will be null if no downloads yet
@@ -75,7 +78,10 @@
                     setTimeout(() => { this.fetch_data() }, 1000);
                 })
             },
+            show_popup(item) {
+			    window.open(item.popup_url, item.id, "width={{ .Config.UI.PopupWidth }},height={{ .Config.UI.PopupHeight }}");
+	    },
         }
     }
 </script>
-{{ end }}
+{{ end }}

web/layout.tmpl

@@ -14,12 +14,54 @@
           overflow:auto;
         }
         footer {
-            padding-top: 50px;
-            font-size: 30%;
+          padding-top: 50px;
+          font-size: 30%;
         }
+        .int-link {
+          text-decoration: none;
+          hover { color: red; }
+        }
+        .state-failed {
+          color: red;
+        }
+        .state-downloading {
+          color: blue;
+        }
+        .state-complete {
+          color: green;
+        }
+        .gropple-config {
+          font-size: 80%;
+        }
+        .gropple-config input.input-long {
+          width: 27em;
+        }
+        .gropple-config button {
+          border-radius: 12px;
+        }
+        .gropple-config button.button-del {
+          background: rgb(202, 60, 60);
+        }
+        .gropple-config button.button-add {
+          background: rgb(60, 200, 60);
+        }
+        .gropple-config .pure-form-message {
+          padding-top: .5em;
+          padding-bottom: 1.5em;
+        }
+        .error {
+          color: red;
+        }
+        .success {
+          color: green;
+        }
+
+        [x-cloak] { display: none !important; }
+
     </style>
 </head>
 <body style="margin:4; padding:4">
+
     {{ template "content" . }}
     <footer>
     Homepage: <a href="https://github.com/tardisx/gropple">https://github.com/tardisx/gropple</a>
@@ -27,4 +69,4 @@
 </body>
     {{ template "js" . }}
 </html>
-{{ end }}
+{{ end }}

web/menu.tmpl

@@ -0,0 +1,15 @@
+    <div class="pure-menu pure-menu-horizontal" style="height: 2em;">
+        <a href="#" class="pure-menu-heading pure-menu-link">gropple</a>
+        <ul class="pure-menu-list">
+            <li class="pure-menu-item">
+                <a href="/" class="pure-menu-link">Home</a>
+            </li>
+            <li class="pure-menu-item">
+                <a href="/config" class="pure-menu-link">Config</a>
+            </li>
+            <li class="pure-menu-item">
+                <a href="https://github.com/tardisx/gropple" class="pure-menu-link">Github</a>
+            </li>
+        </ul>
+    </div>
+    

web/popup.html

@@ -1,8 +1,19 @@
 {{ define "content" }}
-    <div id="layout" class="pure-g pure-u-1" x-data="popup()"  x-init="fetch_data()">
+    <div id="layout" class="pure-g pure-u-1" x-data="popup()" x-init="fetch_data()">
         <h2>Download started</h2>
-        <p>Fetching <tt>{{ .Url }}</tt></p>
+        <p>Fetching <tt>{{ .dl.Url }}</tt></p>
         <table class="pure-table" >
+            <tr>
+                <th>profile</th>
+                <td>
+                    <select x-bind:disabled="profile_chosen" x-on:change="update_profile()" class="pure-input-1-2" x-model="profile_chosen">
+                        <option value="">choose a profile to start</option>
+                    {{ range $i := .config.DownloadProfiles }}
+                        <option>{{ $i.Name }}</option>
+                    {{ end }}
+                    </select>
+                </td>
+            </tr>
             <tr><th>current filename</th><td x-text="filename"></td></tr>
             <tr><th>state</th><td x-text="state"></td></tr>
             <tr><th>progress</th><td x-text="percent"></td></tr>
@@ -19,15 +30,36 @@
 {{ define "js" }}
 <script>
     function popup() {
+        history.replaceState(null, '', ['/fetch/{{ .dl.Id }}'])
         return { 
             eta: '', percent: 0.0, state: '??', filename: '', finished: false, log :'',
+            profile_chosen: null,
+            watch_profile() {
+                this.$watch('profile_chosen', value => this.profile_chosen(value))
+            },
+            update_profile(name) {
+                console.log('you chose name', this.profile_chosen);
+                let op = {
+                   method: 'POST',
+                   body: JSON.stringify({action: 'start', profile: this.profile_chosen}),
+                   headers: { 'Content-Type': 'application/json' }
+                };
+                fetch('/rest/fetch/{{ .dl.Id }}', op)
+                .then(response => response.json())
+                .then(info => {
+                    console.log(info)
+                })
+            },
             fetch_data() {
-                fetch('/fetch/info/{{ .Id }}')
+                fetch('/rest/fetch/{{ .dl.Id }}')
                 .then(response => response.json())
                 .then(info => {
                     this.eta = info.eta;
                     this.percent = info.percent + "%";
                     this.state = info.state;
+                    if (this.state != 'choose profile') {
+                        this.profile_chosen = true;
+                    }
                     this.finished = info.finished;
                     if (info.files && info.files.length > 0) {
                         this.filename = info.files[info.files.length - 1];