Release 1.5

.dockerignore

@@ -0,0 +1,7 @@
+/.venv
+__pycache__
+*.pyc
+/config.yml
+/cache/
+.git
+.gitignore

Dockerfile

@@ -0,0 +1,8 @@
+FROM python:3.13-slim
+
+WORKDIR /app
+COPY . .
+RUN pip install --no-cache-dir -r requirements.txt
+EXPOSE 8080
+
+CMD ["python3", "spothole.py"]

README.md

@@ -33,469 +33,17 @@ see [https://spothole.app/apidocs](https://spothole.app/apidocs) for the API det
 This is a Progressive Web App, so you can also "install" it to your Android or iOS device by accessing it in Chrome or
 Safari respectively, and following the menu-driven process for installing PWAs.
 
-You are more than welcome to use the data and the API that Spothole provides to power your own software. There are many
+You are more than welcome to use the data and the API that Spothole provides to power your own software, to run your
-ways to do this; see below.
+own Spothole server, or to modify it in any way. More details can be found in the following sections:
-
+
-## Embedding Spothole in another website
+* [Embedding Spothole in another website](docs/embedding.md)
-
+* [Writing your own client](docs/clients.md)
-You can embed Spothole's web interface in another website, e.g. for use as part of a ham radio custom dashboard.
+* [Running your own copy](docs/running.md)
-
+  * [nginx reverse proxy configuration](docs/nginx.md)
-URL parameters can be used to trigger an "embedded" mode which hides the headers, footers and settings. In this mode,
+  * [systemd service file](docs/systemd.md)
-you provide configuration for the various filter and display options via additional URL parameters. Any settings that
+  * [Running in Docker](docs/docker.md)
-the user has set for Spothole are ignored. This is so that the embedding site can select, for example, their choice of
+  * [Use of multiple cluster logins](docs/multicluster.md)
-dark mode or SIG filters, which will not impact how Spothole appears when the user accesses it directly. Effectively, it
+* [Modifying the source code](docs/modifying.md)
-becomes separate to their normal Spothole settings.
-
-Setting `embedded` to true is important for the rest of the settings to be applied; otherwise, the user's defaults will
-be used in preference to the URL params.
-
-These are supplied with the URL to the page you want to embed, for example for an embedded version of the band map in
-dark mode, use `https://spothole.app/bands?embedded=true&dark-mode=true`. For an embedded version of the main spots/home
-page in the system light/dark mode, use `https://spothole.app/?embedded=true`. For dark mode showing 70cm TOTA spots
-only, use `https://spothole.app/?embedded=true&dark-mode=true&sig=TOTA&band=70cm`. Providing no URL params causes the
-page to be loaded in the normal way it would when accessed directly in the user's browser.
-
-The supported parameters are as follows. Generally these match the equivalent parameters in the real Spothole API, where
-a mapping exists.
-
-| Name             | Allowed Values          | Default | Example                 | Description                                                                                                                                                           |
-|------------------|-------------------------|---------|-------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `embedded`       | `true`, `false`         | `false` | `?embedded=true`        | Enables embedded mode.                                                                                                                                                |
-| `color-scheme`   | `light`, `dark`, `auto` | `auto`  | `?color-scheme=dark`    | Forces light or dark mode in preference to the operating system default.                                                                                              |
-| `time-zone`      | `UTC`, `local`          | `UTC`   | `?time-zone=local`      | Sets times to be in UTC or local time.                                                                                                                                |
-| `limit`          | 10, 25, 50, 100         | 50      | `?limit=50`             | Sets the number of spots that will be displayed on the main spots page                                                                                                |
-| `limit`          | 25, 50, 100, 200, 500   | 100     | `?limit=100`            | Sets the number of alerts that will be displayed on the alerts page                                                                                                   |
-| `max_age`        | 300, 600, 1800, 3600    | 1800    | `?max_age=1800`         | Sets the maximum age of spots displayed on the map and bands pages, in seconds.                                                                                       |
-| `band`           | Comma-separated list    | (all)   | `?band=20m,40m`         | Sets the list of bands that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface.         |
-| `sig`            | Comma-separated list    | (all)   | `?sig=POTA,SOTA,NO_SIG` | Sets the list of SIGs that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface.          |
-| `source`         | Comma-separated list    | (all)   | `?source=Cluster`       | Sets the list of sources that will be shown on any spot or alert pages. Available options match the labels of the buttons in the standard web interface.              |
-| `mode_type`      | Comma-separated list    | (all)   | `?mode_type=PHONE,CW`   | Sets the list of mode types that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface.    |
-| `dx_continent`   | Comma-separated list    | (all)   | `?dx_continent=NA,SA`   | Sets the list of DX Continents that will be shown on any spot or alert pages. Available options match the labels of the buttons in the standard web interface.        |
-| `de_continent`   | Comma-separated list    | (all)   | `?de_continent=EU`      | Sets the list of DE Continents that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface. |
-| `map-center-lat` | Numeric (decimal)       | (auto)  | `?map-center-lat=51.5`  | Sets the initial latitude of the map centre on the map page. If omitted, the map auto-fits to the loaded spots.                                                       |
-| `map-center-lon` | Numeric (decimal)       | (auto)  | `?map-center-lon=-0.1`  | Sets the initial longitude of the map centre on the map page. If omitted, the map auto-fits to the loaded spots.                                                      |
-| `map-zoom`       | Numeric (integer)       | (auto)  | `?map-zoom=6`           | Sets the initial zoom level of the map on the map page. If omitted, the map auto-fits to the loaded spots.                                                            |
-
-See the comment at the end of the next section regarding reliability and uptime of the "main" server.
-
-## Writing your own client
-
-One of the key strengths of Spothole is that the API is well-defined and open to anyone to use. This means you can build
-your own software that uses data from Spothole.
-
-As well as the main API endpoints to fetch spots and alerts, with various possible query parameters, there are also
-Server-Sent Events (SSE) API endpoints to receive a live feed, plus various utility lookup endpoints for things like
-callsign and park data.
-
-Various approaches exist to writing your own client, but in general:
-
-* Refer to the API docs. These are built on an OpenAPI definition file (`/static/apidocs/openapi.yml`), which you can
-  automatically use to generate a client skeleton using various software.
-* Call the main "spots" or "alerts" API endpoints to get the data you want. For example, your app could call
-  `https://spothole.app/api/v1/spots` once every few minutes. Apply filters if necessary.
-* Call the "options" API to get an idea of which bands, modes etc. the server knows about. You might want to do that
-  first before calling the spots/alerts APIs, to allow you to populate your filters correctly.
-* Refer to the provided HTML/JS interface for a reference on different approaches. For example, the "alerts"/"upcoming"
-  page simply query the main spot API on a timer, whereas the spots, map and bands pages combine this approach with
-  using the Server-Sent Events (SSE) endpoint to update live.
-* Let me know if you get stuck, I'm happy to help.
-
-Please don't hammer the API with an unnecessarily high request rate. For example, Spothole only queries the POTA API
-once every two minutes, so if your client is interested in POTA data there's no need to poll Spothole any more often
-than that.
-
-If you absolutely must be informed within seconds of a spot arriving in Spothole, please use the SSE endpoints instead,
-e.g. `https://spothole.app/api/v1/spots/stream`.
-
-If you want to handle different types of spot or alert differently within your client, please consider making a single
-request to the Spothole API to retrieve all the data, then filtering on your side. For example, call
-`https://spothole.app/api/v1/spots?sig=POTA,SOTA` rather than making two separate calls to
-`https://spothole.app/api/v1/spots?sig=POTA` and `https://spothole.app/api/v1/spots?sig=SOTA`.
-
-Remember, here at Spothole Inc. we offer an industry-standard "five nines" uptime on our server, with our own unique
-twist: we don't tell you which side of the decimal point the nines start! (Translation: This is a hobby project.
-`spothole.app` runs on the same server as my blog and other stuff. It might go down without warning. By all means base
-your own project on data from the main server if you like, but if you want any control over reliability and downtime,
-please run your own copy instead.)
-
-## Running your own copy
-
-If you want to run a copy of Spothole with different configuration settings than the main instance, you can download it
-and run it on your own local machine or server.
-
-You will require Python version 3.8 or later. If you encounter an error about `gdal-config` during the following
-process, you will also need `libgdal-dev` installed.
-
-To download and set up Spothole on a Debian server, run the following commands. Other operating systems will likely be
-similar.
-
-```bash
-git clone ssh://git@git.ianrenton.com/ian/spothole.git
-cd spothole
-python3 -m venv ./.venv
-source .venv/bin/activate
-pip install -r requirements.txt
-deactivate
-cp config-example.yml config.yml
-```
-
-Then edit `config.yml` in your text editor of choice to set up the software as you like it. Mostly, this will involve
-enabling or disabling the various providers of spot and alert data.
-
-By default, all outdoor programme providers are enabled, as is one cluster node and the NG3K DXpedition data. The RBN
-spot providers are turned off by default due to the volume of traffic from CW/RTTY/FT8 skimmers, and the APRS and Packet
-spot providers are off by default on the assumption that Spothole users want a spot with a human at the other end of it,
-but all can be easily re-enabled.
-
-Other parameters you will want to update include the base URL to your instance, and whether you want to serve a full
-web-based DX cluster interface or just the API endpoints for client software to use.
-
-`config.yml` has an entry for a Clublog API key. If provided, this will allow Spothole to retrieve some more information
-about DX spots. The software will work just fine without it, but you may find a few country flags etc. are less accurate
-or missing. Clublog API keys are free, but you'll need to get your own by submitting a helpdesk ticket and explaining
-what you'll use it for. The admin team are happy with the rate of requests made by my Spothole server, so unless you
-change the source code of yours to radically increase the rate of querying Clublog, I'm sure they will be fine with your
-server too.
-
-Once you're happy with the content of `config.yml`, you can proceed to running the software.
-
-To run the software this time and any future times you want to run it directly from the command line:
-
-```bash
-source .venv/bin/activate
-python3 spothole.py
-```
-
-The software can take a few seconds to start up, mostly because it is downloading an updated file to match callsigns to
-countries. This is normal, don't panic!
-
-If you see some errors on startup, check your configuration, e.g. in case you have specified a port for the web server
-that is already in use by something else.
-
-### Multiple cluster nodes with different settings
-
-Dan, S50U has written in with his Spothole cluster settings. He is using a cluster node which provides RBN spots, and
-uses different SSIDs on his callsign to get different settings when logged into the same cluster node. For example:
-
-```
-  -
-    class: "DXCluster"
-    name: "S50CLX"
-    enabled: true
-    host: "s50clx.si"
-    port: 41112
-    login_prompt: "login: "
-    login_callsign: "callsign-10"
-```
-
-Telnet to DXSpider and log in with "callsign-10" and execute the following commands:
-
-`CLEAR/SPOTS ALL` (delete all previous filters)<br/>
-`UNSET/ANN` (stop announce messages)<br/>
-`UNSET/WCY` (stop wcy messages)<br/>
-`UNSET/WWV` (stop wwv messages)<br/>
-`SET/DX` (enable human DX spots)
-
-```
-  -
-    class: "DXCluster"
-    name: "RBN CW"
-    enabled: true
-    host: "s50clx.si"
-    port: 41112
-    login_prompt: "login: "
-    login_callsign: "callsign-11"
-    allow_rbn_spots: true
-    enabled-by-default-in-web-ui: false
-```
-
-Telnet to DXSpider and log in with "callsign-11" and execute the following commands:
-
-`CLEAR/SPOTS ALL` (delete all previous filters)<br/>
-`UNSET/ANN` (stop announce messages)<br/>
-`UNSET/WCY` (stop wcy messages)<br/>
-`UNSET/WWV` (stop wwv messages)<br/>
-`UNSET/DX` (stop human DX spots)<br/>
-`SET/SKIMMER CW` (enable CW RBN spots)
-
-```
-  -
-    class: "DXCluster"
-    name: "RBN RTTY"
-    enabled: true
-    host: "s50clx.si"
-    port: 41112
-    login_prompt: "login: "
-    login_callsign: "callsign-12"
-    allow_rbn_spots: true
-    enabled-by-default-in-web-ui: false
-```
-
-Telnet to DXSpider and log in with "callsign-12" and execute the following commands:
-
-`CLEAR/SPOTS ALL` (delete all previous filters)<br/>
-`UNSET/ANN` (stop announce messages)<br/>
-`UNSET/WCY` (stop wcy messages)<br/>
-`UNSET/WWV` (stop wwv messages)<br/>
-`UNSET/DX` (stop human DX spots)<br/>
-`SET/SKIMMER RTTY` (enable RTTY RBN spots)
-
-```
-  -
-    class: "DXCluster"
-    name: "RBN FT4/8"
-    enabled: true
-    host: "s50clx.si"
-    port: 41112
-    login_prompt: "login: "
-    login_callsign: "callsign-13"
-    allow_rbn_spots: true
-    enabled-by-default-in-web-ui: false
-```
-
-Telnet to DXSpider and log in with "callsign-13" and execute the following commands:
-
-`CLEAR/SPOTS ALL` (delete all previous filters)<br/>
-`UNSET/ANN` (stop announce messages)<br/>
-`UNSET/WCY` (stop wcy messages)<br/>
-`UNSET/WWV` (stop wwv messages)<br/>
-`UNSET/DX` (stop human DX spots)<br/>
-`SET/SKIMMER FT` (enable FT RBN spots)
-
-For each callsign-SSID, we also specify our basic information with commands:
-
-`SET/NAME Spothole10`, Spothole11... etc.<br/>
-`SET/QTH Cerkno`<br/>
-`SET/QRA JN66XD`<br/>
-`SET/HOME S50CLX`
-
-### systemd configuration
-
-If you want Spothole to run automatically on startup on a Linux distribution that uses `systemd`, follow the
-instructions here. For distros that don't use `systemd`, or Windows/OSX/etc., you can find generic instructions for your
-OS online.
-
-Create a file at `/etc/systemd/system/spothole.service`. Give it the following content, adjusting for the user you want
-to run it as and the directory in which you have installed it:
-
-```
-[Unit]
-Description=Spothole
-After=syslog.target network.target
-
-[Service]
-Type=simple
-User=spothole
-WorkingDirectory=/home/spothole/spothole
-ExecStart=/home/spothole/spothole/.venv/bin/python /home/spothole/spothole/spothole.py --serve-in-foreground
-Restart=on-abort
-
-[Install]
-WantedBy=multi-user.target
-```
-
-Run the following:
-
-```bash
-sudo systemctl daemon-reload
-sudo systemctl enable spothole
-sudo systemctl start spothole
-```
-
-Check the service has started up correctly with `sudo journalctl -u spothole -f`.
-
-### nginx Reverse Proxy configuration
-
-Web servers generally serve their pages from port 80. However, it's best not to serve Spothole's web interface directly
-on port 80, as that requires root privileges on a Linux system. It also and prevents us using HTTPS to serve a secure
-site, since Spothole itself doesn't directly support acting as an HTTPS server. The normal solution to this is to use
-a "reverse proxy" setup, where a general web server handles HTTP and HTTP requests (to port 80 & 443 respectively), then
-passes on the request to the back-end application (in this case Spothole). nginx is a common choice for this general web
-server.
-
-To set up nginx as a reverse proxy that sits in front of Spothole, first ensure it's installed e.g.
-`sudo apt install nginx`, and enabled e.g. `sudo systemd enable nginx`.
-
-Create a file at `/etc/nginx/sites-available/` called `spothole`. Give it the following contents, replacing
-`spothole.app` with the domain name on which you want to run Spothole. If you changed the port on which Spothole runs,
-update that on the "proxy_pass" line, and if you installed Spothole somewhere other than `/home/spothole/spothole`,
-adjust the alias location for serving static files.
-
-(The latter section, configuring the nginx server to serve static files directly, improves efficiency because it saves
-Spothole itself from serving JS, CSS etc. files. If you can't do this for some reason, e.g. your nginx and spothole are
-on different computers, you can omit the `location /static/ {}` block.)
-
-```nginx
-server {
-    server_name spothole.app;
-
-    # Global proxy settings
-    proxy_http_version 1.1;
-    proxy_set_header Connection "";
-    proxy_connect_timeout 10s;
-    proxy_buffering on;
-    
-    # Pass on IP address and host information to Spothole, in case logging this information is required
-    proxy_set_header X-Real-IP $remote_addr;
-    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-    proxy_set_header Host $http_host;
-    proxy_set_header X-Forwarded-Proto $scheme;
-
-    # Wellknown area for Lets Encrypt
-    location /.well-known/ {
-        alias /var/www/html/.well-known/;
-    }
-
-    # Load static assets directly from the Spothole static directory
-    location /static/ {
-        alias /home/spothole/spothole/static/;
-        expires 1h;
-        add_header Cache-Control "public, max-age=3600, must-revalidate";
-    }
-    
-    # SSE endpoints
-    location ~ ^/api/v1/(spots|alerts)/stream/? {
-        proxy_pass http://127.0.0.1:8080;
-        
-        # Remove buffering, remove caching, add suitable timeouts for SSE API calls
-        proxy_buffering off;
-        proxy_cache off;
-        proxy_read_timeout 24h;
-        proxy_send_timeout 24h;
-        proxy_set_header X-Accel-Buffering no;
-        add_header Cache-Control no-store always;
-        
-        # Allow cross-origin requests to API
-        proxy_hide_header Access-Control-Allow-Origin;
-        add_header Access-Control-Allow-Origin * always; 
-    }
-
-    # Other API endpoints
-    location /api/ {
-        proxy_pass http://127.0.0.1:8080;
-        
-        # Remove buffering, remove caching, add suitable timeouts for API calls
-        proxy_buffering off;
-        proxy_cache off;
-        proxy_read_timeout 30s;
-        add_header Cache-Control no-store always;
-        
-        # Allow cross-origin requests to API
-        proxy_hide_header Access-Control-Allow-Origin;
-        add_header Access-Control-Allow-Origin * always;
-    }
-
-    # Templated pages
-    location / {
-        proxy_pass http://127.0.0.1:8080;
-        proxy_read_timeout 30s;
-        add_header Cache-Control "no-cache, must-revalidate" always;
-    }
-}
-```
-
-One further change you might want to make to the file above is the `add_header Access-Control-Allow-Origin` statements.
-These are what's used on
-my own Spothole server to make sure that other third-party web-based software can get the data from my instance, and
-applies to any endpoint underneath `/api`. If you want
-*your* Spothole instance to be set up the same way, so that others can write software in JavaScript that can access it,
-leave this intact. But if you want your Spothole instance to only be usable by scripts running on the web server you
-write,
-you can remove these lines. (Note that this doesn't stop other people writing *non-web-based* software that accesses
-your
-Spothole API&mdash;the enforcement of cross-origin headers only happens within the user's browser. If you need to lock
-your
-instance down so that no-one else can access it with *any* software, that's an aspect of nginx or firewall config that
-you will need
-to find help with elsewhere.)
-
-Now, make a symbolic link to enable the site:
-
-```bash
-cd /etc/nginx/sites-enabled
-sudo ln -sf ../sites-available/spothole
-```
-
-Test that your nginx config isn't broken using `nginx -t`. If it works, restart nginx with
-`sudo systemctl restart nginx`.
-
-If you haven't already done so, set up a DNS entry to make sure requests for your domain name end up at the server
-that's running Spothole.
-
-You should now be able to access the web interface by going to the domain from your browser.
-
-Once that's working, [install certbot](https://certbot.eff.org/instructions?ws=nginx&os=snap) onto your server. Run it
-as root, and when prompted pick your domain name from the list. After a few seconds, it should successfully provision a
-certificate and modify your nginx config files automatically. You should then be able to access the site via HTTPS.
-
-## Modifying the source code
-
-Spothole is Public Domain licenced, so you can grab the source code and start modifying it for your own needs.
-Contributions of code back to the main repository are encouraged, but completely optional.
-
-### Code structure
-
-To navigate your way around the source code, this list may help.
-
-*Python back-end code*
-
-* `/core` - Core classes and scripts
-* `/data` - Data storage classes
-* `/spotproviders` - Classes providing spots by accessing the APIs of other services
-* `/alertproviders` - Classes providing alerts by accessing the APIs of other services
-* `/solarconditionsproviders` - Classes providing solar and propagation by accessing the APIs of other services
-* `/server` - Classes for running Spothole's own web server
-
-*Templates*
-
-* `/templates` - Templates used for constructing Spothole's user-targeted HTML pages
-
-*HTML/JS/CSS front-end code*
-
-* `/static` - Root for static files served by the web server. These are all served from a path starting `/static/`.
-* `/static/apidocs` - Contains the OpenAPI spec (`openapi.yml`)
-* `/static/css` - CSS files used by the web front-end
-* `/static/img` - image files used by the web front-end
-* `/static/js` - JavaScript used by the web front-end
-* `/static/vendor` - Third-party libraries (CSS, JS, fonts and images)
-
-*Miscellaneous*
-
-* `/` - Main script (`spothole.py`), pip `requirements.txt`, config, README, etc.
-* `/images` - Image sources
-* `/datafiles` - Local data sources (differentiated from the majority of data files which are loaded from URLs and
-  cached in `/cache`)
-* `/cache` - Directory where static-ish data downloaded from the internet is cached to avoid rapid re-requests, and
-  where spot/alert data is cached so that it survives a software restart. Created on first run.
-
-### Extending the server
-
-Spothole is designed to be easily extensible. If you want to write your own spot provider, for example, simply add a
-module to the `spotproviders` package containing your class. (Currently, in order to be loaded correctly, the module (
-file) name should be the same as the class name, but lower case.)
-
-Your class should extend "SpotProvider"; if it operates by polling an HTTP Server on a timer, it can instead extend "
-HTTPSpotProvider" where some of the work is done for you.
-
-The class will need to implement a constructor that takes in the `provider_config` and provides it to the superclass
-constructor, while also taking any other config parameters it needs.
-
-If you're extending the base `SpotProvider` class, you will need to implement `start()` and `stop()` methods that start
-and stop a separate thread which handles the provider's processing needs. The thread should call `submit()` or
-`submit_batch()` when it has one or more spots to report.
-
-If you're extending the `HTTPSpotProvider` class, you will need to provide a URI to query and an interval to the
-superclass constructor. You'll then need to implement the `http_response_to_spots()` method which is called when new
-data is retrieved. Your implementation should then call `submit()` or `submit_batch()` when it has one or more spots to
-report.
-
-When constructing spots, use the comments in the Spot class and the existing implementations as an example. All
-parameters are optional, but you will at least want to provide a `time` (which must be timezone-aware) and a `dx_call`.
-
-Finally, simply add the appropriate config to the `spot_providers` section of `config.yml`, and your provider should be
-instantiated on startup.
-
-The same approach as above is also used for alert providers.
 
 ## Thanks
 

alertproviders/alert_provider.py

@@ -2,7 +2,7 @@ from datetime import datetime
 
 import pytz
 
-from core.data_store import DATA_STORE
+from core.config import MAX_ALERT_AGE
 
 
 class AlertProvider:
@@ -15,7 +15,14 @@ class AlertProvider:
         self.enabled = provider_config["enabled"]
         self.last_update_time = datetime.min.replace(tzinfo=pytz.UTC)
         self.status = "Not Started" if self.enabled else "Disabled"
-        self._alerts = DATA_STORE.alerts
+        self._alerts = None
+        self._web_server = None
+
+    def setup(self, alerts, web_server):
+        """Set up the provider, e.g. giving it the alert list to work from"""
+
+        self._alerts = alerts
+        self._web_server = web_server
 
     def start(self):
         """Start the provider. This should return immediately after spawning threads to access the remote resources"""
@@ -37,7 +44,10 @@ class AlertProvider:
 
     def _add_alert(self, alert):
         if not alert.expired():
-            self._alerts.set(alert.id, alert)
+            self._alerts.add(alert.id, alert, expire=MAX_ALERT_AGE)
+            # Ping the web server in case we have any SSE connections that need to see this immediately
+            if self._web_server:
+                self._web_server.notify_new_alert(alert)
 
     def stop(self):
         """Stop any threads and prepare for application shutdown"""

core/cache_utils.py

@@ -0,0 +1,27 @@
+import threading
+from datetime import timedelta
+
+from requests_cache import CachedSession
+
+# Cache for "semi-static" data such as the locations of parks, CSVs of reference lists, etc.
+# This has an expiry time of 30 days, so will re-request from the source after that amount
+# of time has passed. This is used throughout Spothole to cache data that does not change
+# rapidly. The ThreadSafeSession construct here protects it against some multithreading
+# contention weirdness we sometimes used to see on startup where the cache was hammered
+# pretty hard. The expanded list of allowable_codes ensures we also cache and return 400-type
+# responses, e.g "this SOTA summit ref doesn't actually exist", to avoid hammering remote
+# servers for data they've told us they can't provide.
+_session = CachedSession("cache/semi_static_url_data_cache", expire_after=timedelta(days=30),
+                         allowable_codes=(200, 400, 401, 403, 404))
+_lock = threading.Lock()
+
+
+class _ThreadSafeSession:
+    """Wraps CachedSession with a lock to prevent concurrent SQLite access across threads."""
+
+    def get(self, *args, **kwargs):
+        with _lock:
+            return _session.get(*args, **kwargs)
+
+
+SEMI_STATIC_URL_DATA_CACHE = _ThreadSafeSession()

core/cleanup.py

@@ -0,0 +1,73 @@
+import logging
+from datetime import datetime
+from threading import Event, Thread
+
+import pytz
+
+
+class CleanupTimer:
+    """Provides a timed cleanup of the spot list."""
+
+    def __init__(self, spots, alerts, web_server, cleanup_interval):
+        """Constructor"""
+
+        self._spots = spots
+        self._alerts = alerts
+        self._web_server = web_server
+        self._cleanup_interval = cleanup_interval
+        self.last_cleanup_time = datetime.min.replace(tzinfo=pytz.UTC)
+        self.status = "Starting"
+        self._thread = None
+        self._stop_event = Event()
+
+    def start(self):
+        """Start the cleanup timer"""
+
+        self._thread = Thread(target=self._run, daemon=True)
+        self._thread.start()
+
+    def stop(self):
+        """Stop any threads and prepare for application shutdown"""
+
+        self._stop_event.set()
+
+    def _run(self):
+        while not self._stop_event.wait(timeout=self._cleanup_interval):
+            self._cleanup()
+
+    def _cleanup(self):
+        """Perform cleanup and reschedule next timer"""
+
+        try:
+            # Perform cleanup via letting the data expire
+            self._spots.expire()
+            self._alerts.expire()
+
+            # Explicitly clean up any spots and alerts that have expired
+            for i in list(self._spots.iterkeys()):
+                try:
+                    spot = self._spots[i]
+                    if spot.expired():
+                        self._spots.delete(i)
+                except KeyError:
+                    # Must have already been deleted, OK with that
+                    pass
+            for i in list(self._alerts.iterkeys()):
+                try:
+                    alert = self._alerts[i]
+                    if alert.expired():
+                        self._alerts.delete(i)
+                except KeyError:
+                    # Must have already been deleted, OK with that
+                    pass
+
+            # Clean up web server SSE spot/alert queues
+            self._web_server.clean_up_sse_queues()
+
+            self.status = "OK"
+            self.last_cleanup_time = datetime.now(pytz.UTC)
+
+        except Exception:
+            self.status = "Error"
+            logging.exception("Exception in Cleanup thread")
+            self._stop_event.wait(timeout=1)

core/constants.py

@@ -3,7 +3,7 @@ from data.band import Band
 from data.sig import SIG
 
 # General software
-SOFTWARE_VERSION = "1.4-pre"
+SOFTWARE_VERSION = "1.5"
 
 # HTTP headers used for spot providers that use HTTP
 HTTP_HEADERS = {"User-Agent": "Spothole v" + SOFTWARE_VERSION + " (operated by " + SERVER_OWNER_CALLSIGN + ")"}

core/data_store.py

@@ -1,69 +0,0 @@
-from pathlib import Path
-
-import diskcache
-
-from core.config import MAX_SPOT_AGE, MAX_ALERT_AGE
-from core.live_data_cache import LiveDataCache
-from data.solar_conditions import SolarConditions
-
-
-class DataStore:
-    """Data caching/storage object. Handles storage of spots, alerts, solar conditions, SIG reference data, and callsign
-    lookup data using different caching strategies for each."""
-
-    def __init__(self):
-        cache_dir = "./cache"
-        self.MAX_SPOT_COUNT = 10000
-        self.MAX_ALERT_COUNT = 10000
-        self.SPOT_ALERT_SNAPSHOT_INTERVAL_SEC = 300
-        self.CALLSIGN_DATA_TTL_SEC = 30 * 24 * 60 * 60
-
-        Path(cache_dir).mkdir(parents=True, exist_ok=True)
-
-        # Special caches for spots and alerts, which have TTL and write snapshots to disk at an interval
-        self.spots = LiveDataCache(maxsize=self.MAX_SPOT_COUNT, ttl=MAX_SPOT_AGE,
-                                   snapshot_dir=cache_dir + "/spots",
-                                   snapshot_interval_sec=self.SPOT_ALERT_SNAPSHOT_INTERVAL_SEC)
-        self.alerts = LiveDataCache(maxsize=self.MAX_ALERT_COUNT, ttl=MAX_ALERT_AGE,
-                                   snapshot_dir=cache_dir + "/alerts",
-                                    snapshot_interval_sec=self.SPOT_ALERT_SNAPSHOT_INTERVAL_SEC)
-
-        # Standard disk cache for solar data and status data, but each cache contains only a single object which we
-        # expose to the wider application
-        self.solar = diskcache.Cache(cache_dir + "/solar")
-        if "solar_conditions" not in self.solar:
-            self.solar.add("solar_conditions", SolarConditions())
-        self.solar_conditions = self.solar.get("solar_conditions")
-        self.status = diskcache.Cache(cache_dir + "/status")
-        if "status_data" not in self.status:
-            self.status.add("status_data", {})
-        self.status_data = self.status.get("status_data")
-
-        # Standard disk caches for SIG ref data. Separate provider threads will repopulate these on a regular basis
-        # but there's no need for a TTL since old data is better than no data.
-        self.sigrefs_wwff = diskcache.Cache(cache_dir + "/sigrefs/wwff")
-        self.sigrefs_siota = diskcache.Cache(cache_dir + "/sigrefs/siota")
-        self.sigrefs_wota = diskcache.Cache(cache_dir + "/sigrefs/wota")
-        self.sigrefs_zlota = diskcache.Cache(cache_dir + "/sigrefs/zlota")
-        self.sigrefs_llota = diskcache.Cache(cache_dir + "/sigrefs/llota")
-        self.sigrefs_dme = diskcache.Cache(cache_dir + "/sigrefs/dme")
-
-        # Standard disk cache for callsign data. This data does have a TTL to trigger an occasional re-lookup.
-        # Old data *is* better than no data, but we can't have a background thread re-looking-up every callsign
-        # we've seen, so we rely on them timing out and this triggering another lookup.
-        self.callsigns = diskcache.Cache(cache_dir + "/callsigns")
-
-    def close(self):
-        self.spots.close()
-        self.alerts.close()
-        self.solar.close()
-        self.status.close()
-        self.sigrefs_wwff.close()
-        self.sigrefs_siota.close()
-        self.sigrefs_wota.close()
-        self.sigrefs_zlota.close()
-        self.sigrefs_llota.close()
-        self.callsigns.close()
-
-# Global object
-DATA_STORE = DataStore()

core/live_data_cache.py

@@ -1,100 +0,0 @@
-import logging
-import threading
-import time
-
-import diskcache
-from cachetools import TTLCache
-
-
-class LiveDataCache:
-    """Cache for spots and alerts. Uses the fast in-memory TTLCache for normal data I/O, including the TTL to enforce
-    maximum lifetime, and adds a separate diskcache to which we can save and load the TTLCache to provide persistence.
-    Also adds thread safety so spots and alerts can come from any thread, and a listener mechanism so the web server
-    can get a callback when new spots/alerts are added, and send them to any SSE clients."""
-
-    def __init__(self, maxsize, ttl, snapshot_dir, snapshot_interval_sec):
-        self._cache = TTLCache(maxsize=maxsize, ttl=ttl)
-        self._lock = threading.Lock()
-        self._ttl = ttl
-        self._listeners = []
-        self._listeners_lock = threading.Lock()
-        self._snapshot_dir = snapshot_dir
-        self._disk_cache = diskcache.Cache(str(snapshot_dir))
-        self._load_snapshot()
-        self._start_periodic_snapshot(snapshot_interval_sec)
-
-    def set(self, key, value):
-        with self._lock:
-            self._cache[key] = value
-
-        # Notify listeners
-        with self._listeners_lock:
-            listeners = list(self._listeners)
-        for callback in listeners:
-            try:
-                callback(value)
-            except Exception:
-                logging.error("Listener raised an exception for key %s", key, exc_info=True)
-
-
-    def get(self, key, default=None):
-        with self._lock:
-            return self._cache.get(key, default)
-
-    def delete(self, key):
-        with self._lock:
-            self._cache.pop(key, None)
-
-    def keys(self):
-        with self._lock:
-            return list(self._cache.keys())
-
-    def values(self):
-        with self._lock:
-            return list(self._cache.values())
-
-    def add_listener(self, callback):
-        """Register callback(value) which will be called whenever a new spot/alert item is added via set(). Used by the
-        web server (via SSEBroadcaster) to send SSE clients an update on every new spot."""
-
-        with self._listeners_lock:
-            self._listeners.append(callback)
-
-    def remove_listener(self, callback):
-        with self._listeners_lock:
-            self._listeners.remove(callback)
-
-    def save_snapshot(self):
-        with self._lock:
-            # Store the time with the data so we can avoid loading anything nxt time that's older than TTL
-            data = [(k, v, time.time()) for k, v in self._cache.items()]
-        try:
-            self._disk_cache.set("snapshot", data)
-        except Exception as e:
-            logging.error("Failed to write snapshot to %s", self._snapshot_dir, e, exc_info=True)
-
-    def _load_snapshot(self):
-        data = self._disk_cache.get("snapshot")
-        if not data:
-            return
-
-        now = time.time()
-        with self._lock:
-            for key, value, saved_at in data:
-                # Only restore entries that would still be within TTL
-                if now - saved_at < self._ttl:
-                    self._cache[key] = value
-        logging.info("Loaded snapshot from %s", self._snapshot_dir)
-
-    def _start_periodic_snapshot(self, interval):
-        def loop():
-            while True:
-                time.sleep(interval)
-                self.save_snapshot()
-
-        t = threading.Thread(target=loop, daemon=True, name=f"snapshot-{self._snapshot_dir}")
-        t.start()
-
-    def close(self):
-        self.save_snapshot()
-        self._disk_cache.close()

core/lookup_helper.py

@@ -14,10 +14,10 @@ from pyhamtools.locator import latlong_to_locator
 from requests.exceptions import ConnectionError, ReadTimeout, ConnectTimeout
 from requests_cache import CachedSession
 
+from core.cache_utils import SEMI_STATIC_URL_DATA_CACHE
 from core.config import config
 from core.constants import BANDS, UNKNOWN_BAND, CW_MODES, PHONE_MODES, DATA_MODES, ALL_MODES, \
     HTTP_HEADERS, HAMQTH_PRG, MODE_ALIASES
-from core.url_data_cache import URL_DATA_CACHE
 
 # QRZ XML field names differ from pyhamtools' normalised names; map them here.
 _QRZ_FIELD_MAP = {
@@ -142,8 +142,8 @@ class LookupHelper:
 
         try:
             logging.info("Downloading Country-files.com cty.plist...")
-            response = URL_DATA_CACHE.get("https://www.country-files.com/cty/cty.plist",
+            response = SEMI_STATIC_URL_DATA_CACHE.get("https://www.country-files.com/cty/cty.plist",
-                                          headers=HTTP_HEADERS)
+                                                      headers=HTTP_HEADERS)
 
             if response.ok:
                 with open(self._country_files_cty_plist_download_location, "w") as f:
@@ -167,7 +167,7 @@ class LookupHelper:
 
         try:
             logging.info("Downloading dxcc.json...")
-            response = URL_DATA_CACHE.get(
+            response = SEMI_STATIC_URL_DATA_CACHE.get(
                 "https://raw.githubusercontent.com/k0swe/dxcc-json/refs/heads/main/dxcc.json",
                 headers=HTTP_HEADERS)
 
@@ -515,7 +515,7 @@ class LookupHelper:
 
         for lookup_call in calls_to_try:
             try:
-                response = URL_DATA_CACHE.get(
+                response = SEMI_STATIC_URL_DATA_CACHE.get(
                     self._qrz_base_url + "?s=" + session_key + "&callsign=" + urllib.parse.quote_plus(lookup_call),
                     headers=HTTP_HEADERS, timeout=10)
                 if response.ok:
@@ -593,7 +593,7 @@ class LookupHelper:
 
         for lookup_call in calls_to_try:
             try:
-                response = URL_DATA_CACHE.get(
+                response = SEMI_STATIC_URL_DATA_CACHE.get(
                     self._hamqth_base_url + "?id=" + session_id + "&callsign=" + urllib.parse.quote_plus(
                         lookup_call) + "&prg=" + HAMQTH_PRG, headers=HTTP_HEADERS)
                 if response.ok:

core/sig_utils.py

@@ -4,16 +4,23 @@ import logging
 from pyhamtools.locator import latlong_to_locator, locator_to_latlong
 from requests.exceptions import ConnectionError, ReadTimeout, ConnectTimeout
 
+from core.cache_utils import SEMI_STATIC_URL_DATA_CACHE
 from core.constants import SIGS, HTTP_HEADERS
-from core.data_store import DATA_STORE
 from core.geo_utils import wab_wai_square_to_lat_lon
-from core.url_data_cache import URL_DATA_CACHE
 
 # Load Spanish municipality data for the DME programme. There's no convenient lookup API for this, so we embed the data
 # file in Spothole and load it on startup.
 with open("datafiles/MUNICIPIOS.csv", encoding="latin-1") as _f:
-    for row in csv.DictReader(_f, delimiter=";"):
+    _DME_INDEX = {row["COD_INE"][:5]: row for row in csv.DictReader(_f, delimiter=";")}
-        DATA_STORE.sigrefs_dme.add(row["COD_INE"][:5], row)
+# Caches for data for the SIGs where we have to download a whole global reference list, rather than looking up a single
+# reference. These get populated from the SEMI_STATIC_URL_DATA_CACHE only if the data actually came
+# live from the internet, to avoid repopulating them every time we pull the same data from the cache.
+_WWFF_INDEX_CACHE = {}
+_SIOTA_INDEX_CACHE = {}
+_WOTA_INDEX_CACHE = {}
+_ZLOTA_INDEX_CACHE = {}
+_LLOTA_INDEX_CACHE = {}
+
 
 def get_ref_regex_for_sig(sig):
     """Utility function to get the regex string for a SIG reference for a named SIG. If no match is found, None will be returned."""
@@ -46,7 +53,7 @@ def populate_sig_ref_info(sig_ref):
     ref_id = sig_ref.id
     try:
         if sig.upper() == "POTA":
-            response = URL_DATA_CACHE.get("https://api.pota.app/park/" + ref_id, headers=HTTP_HEADERS)
+            response = SEMI_STATIC_URL_DATA_CACHE.get("https://api.pota.app/park/" + ref_id, headers=HTTP_HEADERS)
             if response.ok:
                 data = response.json()
                 if data:
@@ -64,7 +71,7 @@ def populate_sig_ref_info(sig_ref):
                 logging.warning("HTTP %d looking up %s ref %s", response.status_code, sig, ref_id)
 
         elif sig.upper() == "SOTA":
-            response = URL_DATA_CACHE.get("https://api-db2.sota.org.uk/api/summits/" + ref_id,
+            response = SEMI_STATIC_URL_DATA_CACHE.get("https://api-db2.sota.org.uk/api/summits/" + ref_id,
                                                       headers=HTTP_HEADERS)
             if response.ok:
                 data = response.json()
@@ -81,7 +88,7 @@ def populate_sig_ref_info(sig_ref):
                 logging.warning("HTTP %d looking up %s ref %s", response.status_code, sig, ref_id)
 
         elif sig.upper() == "WWBOTA":
-            response = URL_DATA_CACHE.get("https://api.wwbota.org/bunkers/" + ref_id,
+            response = SEMI_STATIC_URL_DATA_CACHE.get("https://api.wwbota.org/bunkers/" + ref_id,
                                                       headers=HTTP_HEADERS)
             if response.ok:
                 data = response.json()
@@ -97,7 +104,7 @@ def populate_sig_ref_info(sig_ref):
                 logging.warning("HTTP %d looking up %s ref %s", response.status_code, sig, ref_id)
 
         elif sig.upper() == "GMA" or sig.upper() == "ARLHS" or sig.upper() == "ILLW" or sig.upper() == "WCA" or sig.upper() == "MOTA" or sig.upper() == "IOTA":
-            response = URL_DATA_CACHE.get("https://www.cqgma.org/api/ref/?" + ref_id,
+            response = SEMI_STATIC_URL_DATA_CACHE.get("https://www.cqgma.org/api/ref/?" + ref_id,
                                                       headers=HTTP_HEADERS)
             if response.ok:
                 data = response.json()
@@ -123,14 +130,15 @@ def populate_sig_ref_info(sig_ref):
                 logging.warning("HTTP %d looking up %s ref %s", response.status_code, sig, ref_id)
 
         elif sig.upper() == "WWFF":
-            response = URL_DATA_CACHE.get("https://wwff.co/wwff-data/wwff_directory.csv",
+            response = SEMI_STATIC_URL_DATA_CACHE.get("https://wwff.co/wwff-data/wwff_directory.csv",
                                                       headers=HTTP_HEADERS)
             if response.ok:
-                if not bool(DATA_STORE.sigrefs_wwff) or not response.from_cache:
+                global _WWFF_INDEX_CACHE
+                if not bool(_WWFF_INDEX_CACHE) or not response.from_cache:
                     # New data from WWFF, update our internal map
-                    for row in csv.DictReader(response.content.decode().splitlines()):
+                    _WWFF_INDEX_CACHE = {row["reference"]: row for row in
-                        DATA_STORE.sigrefs_wwff.add(row["reference"], row)
+                                         csv.DictReader(response.content.decode().splitlines())}
-                row = DATA_STORE.sigrefs_wwff.get(ref_id)
+                row = _WWFF_INDEX_CACHE.get(ref_id)
                 if row:
                     sig_ref.name = row["name"] if "name" in row else None
                     sig_ref.url = "https://wwff.co/directory/?showRef=" + ref_id
@@ -144,14 +152,15 @@ def populate_sig_ref_info(sig_ref):
                 logging.warning("HTTP %d looking up %s ref %s", response.status_code, sig, ref_id)
 
         elif sig.upper() == "SIOTA":
-            response = URL_DATA_CACHE.get("https://www.silosontheair.com/data/silos.csv",
+            response = SEMI_STATIC_URL_DATA_CACHE.get("https://www.silosontheair.com/data/silos.csv",
                                                       headers=HTTP_HEADERS)
             if response.ok:
-                if not bool(DATA_STORE.sigrefs_siota) or not response.from_cache:
+                global _SIOTA_INDEX_CACHE
+                if not bool(_SIOTA_INDEX_CACHE) or not response.from_cache:
                     # New data from SIOTA, update our internal map
-                    for row in csv.DictReader(response.content.decode().splitlines()):
+                    _SIOTA_INDEX_CACHE = {row["SILO_CODE"]: row for row in
-                        DATA_STORE.sigrefs_siota.add(row["SILO_CODE"], row)
+                                          csv.DictReader(response.content.decode().splitlines())}
-                row = DATA_STORE.sigrefs_siota.get(ref_id)
+                row = _SIOTA_INDEX_CACHE.get(ref_id)
                 if row:
                     sig_ref.name = row["NAME"] if "NAME" in row else None
                     sig_ref.grid = row["LOCATOR"] if "LOCATOR" in row else None
@@ -163,16 +172,17 @@ def populate_sig_ref_info(sig_ref):
                 logging.warning("HTTP %d looking up %s ref %s", response.status_code, sig, ref_id)
 
         elif sig.upper() == "WOTA":
-            response = URL_DATA_CACHE.get("https://www.wota.org.uk/mapping/data/summits.json",
+            response = SEMI_STATIC_URL_DATA_CACHE.get("https://www.wota.org.uk/mapping/data/summits.json",
                                                       headers=HTTP_HEADERS)
             if response.ok:
                 data = response.json()
                 if data:
-                    if not bool(DATA_STORE.sigrefs_wota) or not response.from_cache:
+                    global _WOTA_INDEX_CACHE
+                    if not bool(_WOTA_INDEX_CACHE) or not response.from_cache:
                         # New data from WOTA, update our internal map
-                        for feature in data.get("features", []):
+                        _WOTA_INDEX_CACHE = {feature["properties"]["wotaId"]: feature for feature in
-                            DATA_STORE.sigrefs_wota.add(feature["properties"]["wotaId"], feature)
+                                             data.get("features", [])}
-                    feature = DATA_STORE.sigrefs_wota.get(ref_id)
+                    feature = _WOTA_INDEX_CACHE.get(ref_id)
                     if feature:
                         sig_ref.name = feature["properties"]["title"]
                         # Fudge WOTA URLs. Outlying fell (LDO) URLs don't match their ID numbers but require 214 to be
@@ -190,24 +200,24 @@ def populate_sig_ref_info(sig_ref):
                 logging.warning("HTTP %d looking up %s ref %s", response.status_code, sig, ref_id)
 
         elif sig.upper() == "ZLOTA":
-            response = URL_DATA_CACHE.get("https://ontheair.nz/assets/assets.json", headers=HTTP_HEADERS)
+            response = SEMI_STATIC_URL_DATA_CACHE.get("https://ontheair.nz/assets/assets.json", headers=HTTP_HEADERS)
             if response.ok:
                 data = response.json()
                 if isinstance(data, list):
-                    if not bool(DATA_STORE.sigrefs_zlota) or not response.from_cache:
+                    global _ZLOTA_INDEX_CACHE
+                    if not bool(_ZLOTA_INDEX_CACHE) or not response.from_cache:
                         # New data from ZLOTA, update our internal map
-                        for ref in data:
+                        _ZLOTA_INDEX_CACHE = {asset["code"]: asset for asset in data}
-                            DATA_STORE.sigrefs_zlota.add(ref["code"], ref)
+                    asset = _ZLOTA_INDEX_CACHE.get(ref_id)
-                    ref = DATA_STORE.sigrefs_zlota.get(ref_id)
+                    if asset:
-                    if ref:
+                        sig_ref.name = asset["name"]
-                        sig_ref.name = ref["name"]
                         sig_ref.url = "https://ontheair.nz/assets/" + ref_id.replace("/", "_")
                         try:
-                            sig_ref.grid = latlong_to_locator(ref["y"], ref["x"], 6)
+                            sig_ref.grid = latlong_to_locator(asset["y"], asset["x"], 6)
                         except:
                             logging.debug("Invalid lat/lon received for reference")
-                            sig_ref.latitude = ref["y"]
+                            sig_ref.latitude = asset["y"]
-                            sig_ref.longitude = ref["x"]
+                            sig_ref.longitude = asset["x"]
                 elif not response.from_cache:
                     logging.warning("Malformed response looking up %s ref %s", sig, ref_id)
             elif not response.from_cache:
@@ -219,16 +229,16 @@ def populate_sig_ref_info(sig_ref):
             sig_ref.url = "https://www.beachesontheair.com/beaches/" + sig_ref.name.lower().replace(" ", "-")
 
         elif sig.upper() == "LLOTA":
-            response = URL_DATA_CACHE.get("https://llota.app/api/public/references",
+            response = SEMI_STATIC_URL_DATA_CACHE.get("https://llota.app/api/public/references",
                                                       headers=HTTP_HEADERS)
             if response.ok:
                 data = response.json()
                 if isinstance(data, list):
-                    if not bool(DATA_STORE.sigrefs_llota) or not response.from_cache:
+                    global _LLOTA_INDEX_CACHE
+                    if not bool(_LLOTA_INDEX_CACHE) or not response.from_cache:
                         # New data from LLOTA, update our internal map
-                        for ref in data:
+                        _LLOTA_INDEX_CACHE = {ref["reference_code"]: ref for ref in data}
-                            DATA_STORE.sigrefs_llota.add(ref["reference_code"], ref)
+                    ref = _LLOTA_INDEX_CACHE.get(ref_id)
-                    ref = DATA_STORE.sigrefs_llota.get(ref_id)
                     if ref:
                         sig_ref.name = str(ref["name"])
                         sig_ref.url = "https://llota.app/list/ref/" + ref_id
@@ -270,7 +280,7 @@ def populate_sig_ref_info(sig_ref):
 
         elif sig.upper() == "DME":
             # Zero-pad to 5 digits to match our source data
-            row = DATA_STORE.sigrefs_dme.get(ref_id.zfill(5))
+            row = _DME_INDEX.get(ref_id.zfill(5))
             if row:
                 sig_ref.name = row["NOMBRE_ACTUAL"] + ", " + row["PROVINCIA"]
                 sig_ref.latitude = float(row["LATITUD_ETRS89_REGCAN95"].replace(",", ".")) if row.get(

core/status_reporter.py

@@ -7,27 +7,31 @@ import pytz
 
 from core.config import SERVER_OWNER_CALLSIGN
 from core.constants import SOFTWARE_VERSION
-from core.data_store import DATA_STORE
 from core.prometheus_metrics_handler import memory_use_gauge, spots_gauge, alerts_gauge
 
 
 class StatusReporter:
     """Provides a timed update of the application's status data."""
 
-    def __init__(self, run_interval, web_server,spot_providers, alert_providers, solar_condition_providers):
+    def __init__(self, status_data, run_interval, web_server, cleanup_timer, spots, spot_providers, alerts,
+                 alert_providers, solar_condition_providers):
         """Constructor"""
 
+        self._status_data = status_data
         self._run_interval = run_interval
         self._web_server = web_server
+        self._cleanup_timer = cleanup_timer
+        self._spots = spots
         self._spot_providers = spot_providers
+        self._alerts = alerts
         self._alert_providers = alert_providers
         self._solar_condition_providers = solar_condition_providers
         self._thread = None
         self._stop_event = Event()
         self._startup_time = datetime.now(pytz.UTC)
 
-        DATA_STORE.status_data["software-version"] = SOFTWARE_VERSION
+        self._status_data["software-version"] = SOFTWARE_VERSION
-        DATA_STORE.status_data["server-owner-callsign"] = SERVER_OWNER_CALLSIGN
+        self._status_data["server-owner-callsign"] = SERVER_OWNER_CALLSIGN
 
     def start(self):
         """Start the reporter thread"""
@@ -51,41 +55,44 @@ class StatusReporter:
     def _report(self):
         """Write status information"""
 
-        DATA_STORE.status_data["uptime"] = (datetime.now(pytz.UTC) - self._startup_time).total_seconds()
+        self._status_data["uptime"] = (datetime.now(pytz.UTC) - self._startup_time).total_seconds()
-        DATA_STORE.status_data["mem_use_mb"] = round(psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024), 3)
+        self._status_data["mem_use_mb"] = round(psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024), 3)
-        DATA_STORE.status_data["num_spots"] = len(DATA_STORE.spots.values())
+        self._status_data["num_spots"] = len(self._spots)
-        DATA_STORE.status_data["num_alerts"] = len(DATA_STORE.alerts.values())
+        self._status_data["num_alerts"] = len(self._alerts)
-        DATA_STORE.status_data["spot_providers"] = list(
+        self._status_data["spot_providers"] = list(
-        map(lambda p: {"name": p.name, "enabled": p.enabled, "status": p.status,
+            map(lambda p: {"name": p.name, "enabled": p.enabled, "status": p.status,
-                       "last_updated": p.last_update_time.replace(
+                           "last_updated": p.last_update_time.replace(
-                           tzinfo=pytz.UTC).timestamp() if p.last_update_time.year > 2000 else 0,
+                               tzinfo=pytz.UTC).timestamp() if p.last_update_time.year > 2000 else 0,
-                       "last_spot": p.last_spot_time.replace(
+                           "last_spot": p.last_spot_time.replace(
-                           tzinfo=pytz.UTC).timestamp() if p.last_spot_time.year > 2000 else 0},
+                               tzinfo=pytz.UTC).timestamp() if p.last_spot_time.year > 2000 else 0},
-            self._spot_providers))
+                self._spot_providers))
-        DATA_STORE.status_data["alert_providers"] = list(
+        self._status_data["alert_providers"] = list(
-        map(lambda p: {"name": p.name, "enabled": p.enabled, "status": p.status,
+            map(lambda p: {"name": p.name, "enabled": p.enabled, "status": p.status,
-                       "last_updated": p.last_update_time.replace(
+                           "last_updated": p.last_update_time.replace(
-                           tzinfo=pytz.UTC).timestamp() if p.last_update_time.year > 2000 else 0},
+                               tzinfo=pytz.UTC).timestamp() if p.last_update_time.year > 2000 else 0},
-            self._alert_providers))
+                self._alert_providers))
-        DATA_STORE.status_data["solar_condition_providers"] = list(
+        self._status_data["solar_condition_providers"] = list(
-        map(lambda p: {"name": p.name, "enabled": p.enabled, "status": p.status,
+            map(lambda p: {"name": p.name, "enabled": p.enabled, "status": p.status,
-                       "last_updated": p.last_update_time.replace(
+                           "last_updated": p.last_update_time.replace(
-                           tzinfo=pytz.UTC).timestamp() if p.last_update_time.year > 2000 else 0},
+                               tzinfo=pytz.UTC).timestamp() if p.last_update_time.year > 2000 else 0},
-            self._solar_condition_providers))
+                self._solar_condition_providers))
-        DATA_STORE.status_data["webserver"] = {"status": self._web_server.web_server_metrics["status"],
+        self._status_data["cleanup"] = {"status": self._cleanup_timer.status,
-                                      "last_api_access": self._web_server.web_server_metrics[
+                                        "last_ran": self._cleanup_timer.last_cleanup_time.replace(
-                                          "last_api_access_time"].replace(
+                                            tzinfo=pytz.UTC).timestamp() if self._cleanup_timer.last_cleanup_time else 0}
-                                          tzinfo=pytz.UTC).timestamp() if self._web_server.web_server_metrics[
+        self._status_data["webserver"] = {"status": self._web_server.web_server_metrics["status"],
-                                          "last_api_access_time"] else 0,
+                                          "last_api_access": self._web_server.web_server_metrics[
-                                      "api_access_count": self._web_server.web_server_metrics["api_access_counter"],
+                                              "last_api_access_time"].replace(
-                                      "last_page_access": self._web_server.web_server_metrics[
+                                              tzinfo=pytz.UTC).timestamp() if self._web_server.web_server_metrics[
-                                          "last_page_access_time"].replace(
+                                              "last_api_access_time"] else 0,
-                                          tzinfo=pytz.UTC).timestamp() if self._web_server.web_server_metrics[
+                                          "api_access_count": self._web_server.web_server_metrics["api_access_counter"],
-                                          "last_page_access_time"] else 0,
+                                          "last_page_access": self._web_server.web_server_metrics[
-                                      "page_access_count": self._web_server.web_server_metrics[
+                                              "last_page_access_time"].replace(
-                                          "page_access_counter"]}
+                                              tzinfo=pytz.UTC).timestamp() if self._web_server.web_server_metrics[
+                                              "last_page_access_time"] else 0,
+                                          "page_access_count": self._web_server.web_server_metrics[
+                                              "page_access_counter"]}
 
         # Update Prometheus metrics
         memory_use_gauge.set(psutil.Process(os.getpid()).memory_info().rss)
-        spots_gauge.set(len(DATA_STORE.spots.values()))
+        spots_gauge.set(len(self._spots))
-        alerts_gauge.set(len(DATA_STORE.alerts.values()))
+        alerts_gauge.set(len(self._alerts))

core/url_data_cache.py

@@ -1,23 +0,0 @@
-import threading
-from datetime import timedelta
-
-from requests_cache import CachedSession
-
-# Cache for "semi-static" data retrieved from a URL. This is a layet of caching in addition to the normal caching of
-# spots, alerts and other data in the DataStore class. Its purpose is to avoid hitting remote endpoints frequently when
-# e.g. restarting Spothole many times during testing.
-_session = CachedSession("cache/semi_static_urls", expire_after=timedelta(days=1),
-                         allowable_codes=(200, 400, 401, 403, 404))
-_lock = threading.Lock()
-
-
-class _ThreadSafeSession:
-    """Wraps CachedSession with a lock to prevent concurrent SQLite access across threads. This allows a single object
-    to be used freely across the application."""
-
-    def get(self, *args, **kwargs):
-        with _lock:
-            return _session.get(*args, **kwargs)
-
-# Global object
-URL_DATA_CACHE = _ThreadSafeSession()

core/utils.py

@@ -5,4 +5,14 @@ def safe_json_dumps(obj):
     """Safe version of json.dumps that also converts objects to dicts so they can be output, and ignores NaN floats
     which are invalid in JSON."""
 
-    return simplejson.dumps(obj, ensure_ascii=False, ignore_nan=True, default=lambda o: o.__dict__)
+    return simplejson.dumps(obj, ensure_ascii=False, ignore_nan=True, default=lambda o: o.__dict__)
+
+
+def empty_queue(q):
+    """Empty a queue"""
+
+    while not q.empty():
+        try:
+            q.get_nowait()
+        except:
+            break

data/spot.py

@@ -325,6 +325,19 @@ class Spot:
                         self.propagation_mode = mode_tag
                         logging.info("Seen a new propagation mode tag not yet in the system: %s", mode_tag)
 
+        # Parse "de_grid -> dx_grid" structures from the comment
+        if self.comment:
+            grid_mode_grid_match = re.search(
+                r'\b([A-Ra-r]{2}\d{2}(?:[A-Xa-x]{2}(?:\d{2})?)?)\s*->\s*([A-Ra-r]{2}\d{2}(?:[A-Xa-x]{2}(?:\d{2})?)?)\b',
+                self.comment)
+            if grid_mode_grid_match:
+                # regex matches, so extract grids:
+                if not self.dx_grid:
+                    self.dx_grid = grid_mode_grid_match.group(1).upper()
+                    self.dx_location_source = "SPOT"
+                if not self.de_grid:
+                    self.de_grid = grid_mode_grid_match.group(2).upper()
+
         # DX Grid to lat/lon and vice versa in case one is missing
         if self.dx_grid and not self.dx_latitude:
             try:

docs/clients.md

@@ -0,0 +1,39 @@
+## Writing your own client
+
+One of the key strengths of Spothole is that the API is well-defined and open to anyone to use. This means you can build
+your own software that uses data from Spothole.
+
+As well as the main API endpoints to fetch spots and alerts, with various possible query parameters, there are also
+Server-Sent Events (SSE) API endpoints to receive a live feed, plus various utility lookup endpoints for things like
+callsign and park data.
+
+Various approaches exist to writing your own client, but in general:
+
+* Refer to the API docs. These are built on an OpenAPI definition file (`/static/apidocs/openapi.yml`), which you can
+  automatically use to generate a client skeleton using various software.
+* Call the main "spots" or "alerts" API endpoints to get the data you want. For example, your app could call
+  `https://spothole.app/api/v1/spots` once every few minutes. Apply filters if necessary.
+* Call the "options" API to get an idea of which bands, modes etc. the server knows about. You might want to do that
+  first before calling the spots/alerts APIs, to allow you to populate your filters correctly.
+* Refer to the provided HTML/JS interface for a reference on different approaches. For example, the "alerts"/"upcoming"
+  page simply query the main spot API on a timer, whereas the spots, map and bands pages combine this approach with
+  using the Server-Sent Events (SSE) endpoint to update live.
+* Let me know if you get stuck, I'm happy to help.
+
+Please don't hammer the API with an unnecessarily high request rate. For example, Spothole only queries the POTA API
+once every two minutes, so if your client is interested in POTA data there's no need to poll Spothole any more often
+than that.
+
+If you absolutely must be informed within seconds of a spot arriving in Spothole, please use the SSE endpoints instead,
+e.g. `https://spothole.app/api/v1/spots/stream`.
+
+If you want to handle different types of spot or alert differently within your client, please consider making a single
+request to the Spothole API to retrieve all the data, then filtering on your side. For example, call
+`https://spothole.app/api/v1/spots?sig=POTA,SOTA` rather than making two separate calls to
+`https://spothole.app/api/v1/spots?sig=POTA` and `https://spothole.app/api/v1/spots?sig=SOTA`.
+
+Remember, here at Spothole Inc. we offer an industry-standard "five nines" uptime on our server, with our own unique
+twist: we don't tell you which side of the decimal point the nines start! (Translation: This is a hobby project.
+`spothole.app` runs on the same server as my blog and other stuff. It might go down without warning. By all means base
+your own project on data from the main server if you like, but if you want any control over reliability and downtime,
+please run your own copy instead.)

docs/docker.md

@@ -0,0 +1,150 @@
+## Running using Docker
+
+Spothole comes with a Docker configuration to make it easy to run it in a containerised environment. To set it up using
+Docker, the easiest way is to use a Docker Compose file. Create a new directory such as `/opt/docker/spothole` and
+create a `compose.yaml` file inside it with the following contents:
+
+```yaml
+services:
+  spothole:
+    container_name: spothole
+    build:
+      context: https://git.ianrenton.com/ian/spothole.git#main
+    restart: unless-stopped
+    ports:
+      - "8080:8080"
+    volumes:
+      - ./config.yml:/app/config.yml
+      - ./cache:/app/cache
+```
+
+You can replace `#main` with any other branch or tag reference, for example `#1.5` to pin the build to tagged version
+1.5.
+
+Save the file. You will still need to create a copy of `config-example.yml` and name it `config.yml`, though with the
+Docker setup nothing has actually been downloaded yet, so you will have to copy the example from the repository some
+other way, e.g. [from the repo in a web browser](https://git.ianrenton.com/ian/spothole/src/branch/main/config-example.yml).
+
+With that in place, run `docker compose up` and you should be good to go. To detach, press `d` or run the command with
+the `-d` flag.
+
+### nginx Reverse Proxy with Docker
+
+In a containerised setup, it's typical to run an nginx reverse proxy in one container, alongside certbot for renewal
+of HTTPS certificates, and then applications like Spothole in a separate container. In this case, there are a couple of
+variations of the docker compose file above, and the nginx reverse proxy configuration covered [here](./nginx.md), that
+you will want to make.
+
+1. A port mapping is no longer required in the docker compose file; nginx will access into the docker container directly
+   on e.g. `http://spothole:8080`
+2. Spothole and nginx will need to be on the same docker network.
+
+So your `compose.yaml` might look like this:
+
+```yaml
+services:
+  spothole:
+    container_name: spothole
+    build:
+      context: https://git.ianrenton.com/ian/spothole.git#main
+    restart: unless-stopped
+    networks:
+      - docker-network
+    volumes:
+      - ./config.yml:/app/config.yml
+      - ./cache:/app/cache
+
+networks:
+  docker-network:
+    external: true
+```
+
+In your nginx site configuration, you'll want to refer to the Spothole container directly, and drop the block that
+allows nginx to access static files directly, as these will be inaccessible in another container. So you may end up
+with something like:
+
+```nginx
+server {
+    server_name spothole.app;
+
+    # Global proxy settings
+    proxy_http_version 1.1;
+    proxy_set_header Connection "";
+    proxy_connect_timeout 10s;
+    proxy_buffering on;
+    
+    # Pass on IP address and host information to Spothole, in case logging this information is required
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    proxy_set_header Host $http_host;
+    proxy_set_header X-Forwarded-Proto $scheme;
+
+    # Wellknown area for Lets Encrypt
+    location /.well-known/acme-challenge/ {
+        root /var/www/certbot;
+    }
+    
+    # SSE endpoints
+    location ~ ^/api/v1/(spots|alerts)/stream/? {
+        proxy_pass http://spothole:8080;
+        
+        # Remove buffering, remove caching, add suitable timeouts for SSE API calls
+        proxy_buffering off;
+        proxy_cache off;
+        proxy_read_timeout 24h;
+        proxy_send_timeout 24h;
+        proxy_set_header X-Accel-Buffering no;
+        add_header Cache-Control no-store always;
+        
+        # Allow cross-origin requests to API
+        proxy_hide_header Access-Control-Allow-Origin;
+        add_header Access-Control-Allow-Origin * always; 
+    }
+
+    # Other API endpoints
+    location /api/ {
+        proxy_pass http://spothole:8080;
+        
+        # Remove buffering, remove caching, add suitable timeouts for API calls
+        proxy_buffering off;
+        proxy_cache off;
+        proxy_read_timeout 30s;
+        add_header Cache-Control no-store always;
+        
+        # Allow cross-origin requests to API
+        proxy_hide_header Access-Control-Allow-Origin;
+        add_header Access-Control-Allow-Origin * always;
+    }
+
+    # Templated pages and static assets
+    location / {
+        proxy_pass http://spothole:8080;
+        proxy_read_timeout 30s;
+        add_header Cache-Control "no-cache, must-revalidate" always;
+    }
+    
+    listen 443 ssl;
+    listen [::]:443 ssl;
+
+    ssl_certificate /etc/letsencrypt/live/spothole.app/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/spothole.app/privkey.pem;
+    include /etc/letsencrypt/options-ssl-nginx.conf;
+    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
+}
+
+
+server {
+    if ($host = spothole.app) {
+        return 301 https://$host$request_uri;
+    }
+
+    server_name spothole.app;
+    listen 80;
+    listen [::]:80;
+    return 404;
+}
+```
+
+If desired, you could even change the port on which Spothole runs from 8080 to a plain 80, in which case your
+`proxy_pass` statements could drop the `:8080` suffix. Since Spothole is in a container, it can serve HTTP on port 80
+if desired, because it doesn't conflict with the host system.

docs/embedding.md

@@ -0,0 +1,41 @@
+## Embedding Spothole in another website
+
+You can embed Spothole's web interface in another website, e.g. for use as part of a ham radio custom dashboard.
+
+URL parameters can be used to trigger an "embedded" mode which hides the headers, footers and settings. In this mode,
+you provide configuration for the various filter and display options via additional URL parameters. Any settings that
+the user has set for Spothole are ignored. This is so that the embedding site can select, for example, their choice of
+dark mode or SIG filters, which will not impact how Spothole appears when the user accesses it directly. Effectively, it
+becomes separate to their normal Spothole settings.
+
+Setting `embedded` to true is important for the rest of the settings to be applied; otherwise, the user's defaults will
+be used in preference to the URL params.
+
+These are supplied with the URL to the page you want to embed, for example for an embedded version of the band map in
+dark mode, use `https://spothole.app/bands?embedded=true&dark-mode=true`. For an embedded version of the main spots/home
+page in the system light/dark mode, use `https://spothole.app/?embedded=true`. For dark mode showing 70cm TOTA spots
+only, use `https://spothole.app/?embedded=true&dark-mode=true&sig=TOTA&band=70cm`. Providing no URL params causes the
+page to be loaded in the normal way it would when accessed directly in the user's browser.
+
+The supported parameters are as follows. Generally these match the equivalent parameters in the real Spothole API, where
+a mapping exists.
+
+| Name             | Allowed Values          | Default | Example                 | Description                                                                                                                                                           |
+|------------------|-------------------------|---------|-------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `embedded`       | `true`, `false`         | `false` | `?embedded=true`        | Enables embedded mode.                                                                                                                                                |
+| `color-scheme`   | `light`, `dark`, `auto` | `auto`  | `?color-scheme=dark`    | Forces light or dark mode in preference to the operating system default.                                                                                              |
+| `time-zone`      | `UTC`, `local`          | `UTC`   | `?time-zone=local`      | Sets times to be in UTC or local time.                                                                                                                                |
+| `limit`          | 10, 25, 50, 100         | 50      | `?limit=50`             | Sets the number of spots that will be displayed on the main spots page                                                                                                |
+| `limit`          | 25, 50, 100, 200, 500   | 100     | `?limit=100`            | Sets the number of alerts that will be displayed on the alerts page                                                                                                   |
+| `max_age`        | 300, 600, 1800, 3600    | 1800    | `?max_age=1800`         | Sets the maximum age of spots displayed on the map and bands pages, in seconds.                                                                                       |
+| `band`           | Comma-separated list    | (all)   | `?band=20m,40m`         | Sets the list of bands that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface.         |
+| `sig`            | Comma-separated list    | (all)   | `?sig=POTA,SOTA,NO_SIG` | Sets the list of SIGs that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface.          |
+| `source`         | Comma-separated list    | (all)   | `?source=Cluster`       | Sets the list of sources that will be shown on any spot or alert pages. Available options match the labels of the buttons in the standard web interface.              |
+| `mode_type`      | Comma-separated list    | (all)   | `?mode_type=PHONE,CW`   | Sets the list of mode types that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface.    |
+| `dx_continent`   | Comma-separated list    | (all)   | `?dx_continent=NA,SA`   | Sets the list of DX Continents that will be shown on any spot or alert pages. Available options match the labels of the buttons in the standard web interface.        |
+| `de_continent`   | Comma-separated list    | (all)   | `?de_continent=EU`      | Sets the list of DE Continents that will be shown on the spots, bands and map pages. Available options match the labels of the buttons in the standard web interface. |
+| `map-center-lat` | Numeric (decimal)       | (auto)  | `?map-center-lat=51.5`  | Sets the initial latitude of the map centre on the map page. If omitted, the map auto-fits to the loaded spots.                                                       |
+| `map-center-lon` | Numeric (decimal)       | (auto)  | `?map-center-lon=-0.1`  | Sets the initial longitude of the map centre on the map page. If omitted, the map auto-fits to the loaded spots.                                                      |
+| `map-zoom`       | Numeric (integer)       | (auto)  | `?map-zoom=6`           | Sets the initial zoom level of the map on the map page. If omitted, the map auto-fits to the loaded spots.                                                            |
+
+See the comment at the end of the next section regarding reliability and uptime of the "main" server.

docs/modifying.md

@@ -0,0 +1,69 @@
+## Modifying the source code
+
+Spothole is Public Domain licenced, so you can grab the source code and start modifying it for your own needs.
+Contributions of code back to the main repository are encouraged, but completely optional.
+
+### Code structure
+
+To navigate your way around the source code, this list may help.
+
+*Python back-end code*
+
+* `/core` - Core classes and scripts
+* `/data` - Data storage classes
+* `/spotproviders` - Classes providing spots by accessing the APIs of other services
+* `/alertproviders` - Classes providing alerts by accessing the APIs of other services
+* `/solarconditionsproviders` - Classes providing solar and propagation by accessing the APIs of other services
+* `/server` - Classes for running Spothole's own web server
+
+*Templates*
+
+* `/templates` - Templates used for constructing Spothole's user-targeted HTML pages
+
+*HTML/JS/CSS front-end code*
+
+* `/static` - Root for static files served by the web server. These are all served from a path starting `/static/`.
+* `/static/apidocs` - Contains the OpenAPI spec (`openapi.yml`)
+* `/static/css` - CSS files used by the web front-end
+* `/static/img` - image files used by the web front-end
+* `/static/js` - JavaScript used by the web front-end
+* `/static/vendor` - Third-party libraries (CSS, JS, fonts and images)
+
+*Miscellaneous*
+
+* `/` - Main script (`spothole.py`), pip `requirements.txt`, config, README, etc.
+* `/docs` - Documentation
+* `/images` - Image sources
+* `/datafiles` - Local data sources (differentiated from the majority of data files which are loaded from URLs and
+  cached in `/cache`)
+* `/cache` - Directory where static-ish data downloaded from the internet is cached to avoid rapid re-requests, and
+  where spot/alert data is cached so that it survives a software restart. Created on first run.
+
+### Extending the server
+
+Spothole is designed to be easily extensible. If you want to write your own spot provider, for example, simply add a
+module to the `spotproviders` package containing your class. (Currently, in order to be loaded correctly, the module (
+file) name should be the same as the class name, but lower case.)
+
+Your class should extend "SpotProvider"; if it operates by polling an HTTP Server on a timer, it can instead extend "
+HTTPSpotProvider" where some of the work is done for you.
+
+The class will need to implement a constructor that takes in the `provider_config` and provides it to the superclass
+constructor, while also taking any other config parameters it needs.
+
+If you're extending the base `SpotProvider` class, you will need to implement `start()` and `stop()` methods that start
+and stop a separate thread which handles the provider's processing needs. The thread should call `submit()` or
+`submit_batch()` when it has one or more spots to report.
+
+If you're extending the `HTTPSpotProvider` class, you will need to provide a URI to query and an interval to the
+superclass constructor. You'll then need to implement the `http_response_to_spots()` method which is called when new
+data is retrieved. Your implementation should then call `submit()` or `submit_batch()` when it has one or more spots to
+report.
+
+When constructing spots, use the comments in the Spot class and the existing implementations as an example. All
+parameters are optional, but you will at least want to provide a `time` (which must be timezone-aware) and a `dx_call`.
+
+Finally, simply add the appropriate config to the `spot_providers` section of `config.yml`, and your provider should be
+instantiated on startup.
+
+The same approach as above is also used for alert providers.

docs/multicluster.md

@@ -0,0 +1,96 @@
+## Multiple cluster nodes with different settings
+
+Dan, S50U has written in with his Spothole cluster settings. He is using a cluster node which provides RBN spots, and
+uses different SSIDs on his callsign to get different settings when logged into the same cluster node. For example:
+
+```
+  -
+    class: "DXCluster"
+    name: "S50CLX"
+    enabled: true
+    host: "s50clx.si"
+    port: 41112
+    login_prompt: "login: "
+    login_callsign: "callsign-10"
+```
+
+Telnet to DXSpider and log in with "callsign-10" and execute the following commands:
+
+`CLEAR/SPOTS ALL` (delete all previous filters)<br/>
+`UNSET/ANN` (stop announce messages)<br/>
+`UNSET/WCY` (stop wcy messages)<br/>
+`UNSET/WWV` (stop wwv messages)<br/>
+`SET/DX` (enable human DX spots)
+
+```
+  -
+    class: "DXCluster"
+    name: "RBN CW"
+    enabled: true
+    host: "s50clx.si"
+    port: 41112
+    login_prompt: "login: "
+    login_callsign: "callsign-11"
+    allow_rbn_spots: true
+    enabled-by-default-in-web-ui: false
+```
+
+Telnet to DXSpider and log in with "callsign-11" and execute the following commands:
+
+`CLEAR/SPOTS ALL` (delete all previous filters)<br/>
+`UNSET/ANN` (stop announce messages)<br/>
+`UNSET/WCY` (stop wcy messages)<br/>
+`UNSET/WWV` (stop wwv messages)<br/>
+`UNSET/DX` (stop human DX spots)<br/>
+`SET/SKIMMER CW` (enable CW RBN spots)
+
+```
+  -
+    class: "DXCluster"
+    name: "RBN RTTY"
+    enabled: true
+    host: "s50clx.si"
+    port: 41112
+    login_prompt: "login: "
+    login_callsign: "callsign-12"
+    allow_rbn_spots: true
+    enabled-by-default-in-web-ui: false
+```
+
+Telnet to DXSpider and log in with "callsign-12" and execute the following commands:
+
+`CLEAR/SPOTS ALL` (delete all previous filters)<br/>
+`UNSET/ANN` (stop announce messages)<br/>
+`UNSET/WCY` (stop wcy messages)<br/>
+`UNSET/WWV` (stop wwv messages)<br/>
+`UNSET/DX` (stop human DX spots)<br/>
+`SET/SKIMMER RTTY` (enable RTTY RBN spots)
+
+```
+  -
+    class: "DXCluster"
+    name: "RBN FT4/8"
+    enabled: true
+    host: "s50clx.si"
+    port: 41112
+    login_prompt: "login: "
+    login_callsign: "callsign-13"
+    allow_rbn_spots: true
+    enabled-by-default-in-web-ui: false
+```
+
+Telnet to DXSpider and log in with "callsign-13" and execute the following commands:
+
+`CLEAR/SPOTS ALL` (delete all previous filters)<br/>
+`UNSET/ANN` (stop announce messages)<br/>
+`UNSET/WCY` (stop wcy messages)<br/>
+`UNSET/WWV` (stop wwv messages)<br/>
+`UNSET/DX` (stop human DX spots)<br/>
+`SET/SKIMMER FT` (enable FT RBN spots)
+
+For each callsign-SSID, we also specify our basic information with commands:
+
+`SET/NAME Spothole10`, Spothole11... etc.<br/>
+`SET/QTH Cerkno`<br/>
+`SET/QRA JN66XD`<br/>
+`SET/HOME S50CLX`

docs/nginx.md

@@ -0,0 +1,123 @@
+## nginx Reverse Proxy configuration
+
+Web servers generally serve their pages from port 80. However, it's best not to serve Spothole's web interface directly
+on port 80, as that requires root privileges on a Linux system. It also and prevents us using HTTPS to serve a secure
+site, since Spothole itself doesn't directly support acting as an HTTPS server. The normal solution to this is to use
+a "reverse proxy" setup, where a general web server handles HTTP and HTTP requests (to port 80 & 443 respectively), then
+passes on the request to the back-end application (in this case Spothole). nginx is a common choice for this general web
+server.
+
+To set up nginx as a reverse proxy that sits in front of Spothole, first ensure it's installed e.g.
+`sudo apt install nginx`, and enabled e.g. `sudo systemd enable nginx`.
+
+Create a file at `/etc/nginx/sites-available/` called `spothole`. Give it the following contents, replacing
+`spothole.app` with the domain name on which you want to run Spothole. If you changed the port on which Spothole runs,
+update that on the "proxy_pass" line, and if you installed Spothole somewhere other than `/home/spothole/spothole`,
+adjust the alias location for serving static files.
+
+(The latter section, configuring the nginx server to serve static files directly, improves efficiency because it saves
+Spothole itself from serving JS, CSS etc. files. If you can't do this for some reason, e.g. your nginx and spothole are
+on different computers, you can omit the `location /static/ {}` block.)
+
+```nginx
+server {
+    server_name spothole.app;
+
+    # Global proxy settings
+    proxy_http_version 1.1;
+    proxy_set_header Connection "";
+    proxy_connect_timeout 10s;
+    proxy_buffering on;
+    
+    # Pass on IP address and host information to Spothole, in case logging this information is required
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    proxy_set_header Host $http_host;
+    proxy_set_header X-Forwarded-Proto $scheme;
+
+    # Wellknown area for Lets Encrypt
+    location /.well-known/ {
+        alias /var/www/html/.well-known/;
+    }
+
+    # Load static assets directly from the Spothole static directory
+    location /static/ {
+        alias /home/spothole/spothole/static/;
+        expires 1h;
+        add_header Cache-Control "public, max-age=3600, must-revalidate";
+    }
+    
+    # SSE endpoints
+    location ~ ^/api/v1/(spots|alerts)/stream/? {
+        proxy_pass http://127.0.0.1:8080;
+        
+        # Remove buffering, remove caching, add suitable timeouts for SSE API calls
+        proxy_buffering off;
+        proxy_cache off;
+        proxy_read_timeout 24h;
+        proxy_send_timeout 24h;
+        proxy_set_header X-Accel-Buffering no;
+        add_header Cache-Control no-store always;
+        
+        # Allow cross-origin requests to API
+        proxy_hide_header Access-Control-Allow-Origin;
+        add_header Access-Control-Allow-Origin * always; 
+    }
+
+    # Other API endpoints
+    location /api/ {
+        proxy_pass http://127.0.0.1:8080;
+        
+        # Remove buffering, remove caching, add suitable timeouts for API calls
+        proxy_buffering off;
+        proxy_cache off;
+        proxy_read_timeout 30s;
+        add_header Cache-Control no-store always;
+        
+        # Allow cross-origin requests to API
+        proxy_hide_header Access-Control-Allow-Origin;
+        add_header Access-Control-Allow-Origin * always;
+    }
+
+    # Templated pages
+    location / {
+        proxy_pass http://127.0.0.1:8080;
+        proxy_read_timeout 30s;
+        add_header Cache-Control "no-cache, must-revalidate" always;
+    }
+}
+```
+
+One further change you might want to make to the file above is the `add_header Access-Control-Allow-Origin` statements.
+These are what's used on
+my own Spothole server to make sure that other third-party web-based software can get the data from my instance, and
+applies to any endpoint underneath `/api`. If you want
+*your* Spothole instance to be set up the same way, so that others can write software in JavaScript that can access it,
+leave this intact. But if you want your Spothole instance to only be usable by scripts running on the web server you
+write,
+you can remove these lines. (Note that this doesn't stop other people writing *non-web-based* software that accesses
+your
+Spothole API—the enforcement of cross-origin headers only happens within the user's browser. If you need to lock
+your
+instance down so that no-one else can access it with *any* software, that's an aspect of nginx or firewall config that
+you will need
+to find help with elsewhere.)
+
+Now, make a symbolic link to enable the site:
+
+```bash
+cd /etc/nginx/sites-enabled
+sudo ln -sf ../sites-available/spothole
+```
+
+Test that your nginx config isn't broken using `nginx -t`. If it works, restart nginx with
+`sudo systemctl restart nginx`.
+
+If you haven't already done so, set up a DNS entry to make sure requests for your domain name end up at the server
+that's running Spothole.
+
+You should now be able to access the web interface by going to the domain from your browser.
+
+Once that's working, [install certbot](https://certbot.eff.org/instructions?ws=nginx&os=snap) onto your server. Run it
+as root, and when prompted pick your domain name from the list. After a few seconds, it should successfully provision a
+certificate and modify your nginx config files automatically. You should then be able to access the site via HTTPS.

docs/running.md

@@ -0,0 +1,53 @@
+## Running your own copy
+
+If you want to run a copy of Spothole with different configuration settings than the main instance, you can download it
+and run it on your own local machine or server.
+
+You will require Python version 3.8 or later. If you encounter an error about `gdal-config` during the following
+process, you will also need `libgdal-dev` installed.
+
+To download and set up Spothole on a Debian server, run the following commands. Other operating systems will likely be
+similar.
+
+```bash
+git clone ssh://git@git.ianrenton.com/ian/spothole.git
+cd spothole
+python3 -m venv ./.venv
+source .venv/bin/activate
+pip install -r requirements.txt
+deactivate
+cp config-example.yml config.yml
+```
+
+Then edit `config.yml` in your text editor of choice to set up the software as you like it. Mostly, this will involve
+enabling or disabling the various providers of spot and alert data.
+
+By default, all outdoor programme providers are enabled, as is one cluster node and the NG3K DXpedition data. The RBN
+spot providers are turned off by default due to the volume of traffic from CW/RTTY/FT8 skimmers, and the APRS and Packet
+spot providers are off by default on the assumption that Spothole users want a spot with a human at the other end of it,
+but all can be easily re-enabled.
+
+Other parameters you will want to update include the base URL to your instance, and whether you want to serve a full
+web-based DX cluster interface or just the API endpoints for client software to use.
+
+`config.yml` has an entry for a Clublog API key. If provided, this will allow Spothole to retrieve some more information
+about DX spots. The software will work just fine without it, but you may find a few country flags etc. are less accurate
+or missing. Clublog API keys are free, but you'll need to get your own by submitting a helpdesk ticket and explaining
+what you'll use it for. The admin team are happy with the rate of requests made by my Spothole server, so unless you
+change the source code of yours to radically increase the rate of querying Clublog, I'm sure they will be fine with your
+server too.
+
+Once you're happy with the content of `config.yml`, you can proceed to running the software.
+
+To run the software this time and any future times you want to run it directly from the command line:
+
+```bash
+source .venv/bin/activate
+python3 spothole.py
+```
+
+The software can take a few seconds to start up, mostly because it is downloading an updated file to match callsigns to
+countries. This is normal, don't panic!
+
+If you see some errors on startup, check your configuration, e.g. in case you have specified a port for the web server
+that is already in use by something else.

docs/systemd.md

@@ -0,0 +1,34 @@
+## systemd configuration
+
+If you want Spothole to run automatically on startup on a Linux distribution that uses `systemd`, follow the
+instructions here. For distros that don't use `systemd`, or Windows/OSX/etc., you can find generic instructions for your
+OS online.
+
+Create a file at `/etc/systemd/system/spothole.service`. Give it the following content, adjusting for the user you want
+to run it as and the directory in which you have installed it:
+
+```
+[Unit]
+Description=Spothole
+After=syslog.target network.target
+
+[Service]
+Type=simple
+User=spothole
+WorkingDirectory=/home/spothole/spothole
+ExecStart=/home/spothole/spothole/.venv/bin/python /home/spothole/spothole/spothole.py --serve-in-foreground
+Restart=on-abort
+
+[Install]
+WantedBy=multi-user.target
+```
+
+Run the following:
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl enable spothole
+sudo systemctl start spothole
+```
+
+Check the service has started up correctly with `sudo journalctl -u spothole -f`.

requirements.txt

@@ -17,5 +17,4 @@ websocket-client~=1.8.0
 tornado~=6.4.2
 tornado_eventsource~=3.0.0
 geopandas~=0.13.2
-simplejson~=4.1.1
+simplejson~=4.1.1
-cachetools~=7.1.6

server/handlers/api/addspot.py

@@ -8,7 +8,7 @@ import tornado
 from tornado import httputil
 from tornado.web import Application
 
-from core.config import ALLOW_SPOTTING
+from core.config import ALLOW_SPOTTING, MAX_SPOT_AGE
 from core.constants import UNKNOWN_BAND
 from core.lookup_helper import infer_band_from_freq
 from core.prometheus_metrics_handler import api_requests_counter
@@ -119,7 +119,7 @@ class APISpotHandler(tornado.web.RequestHandler):
             # infer missing data, and add it to our database.
             spot.source = "API"
             spot.infer_missing()
-            self._spots.set(spot.id, spot)
+            self._spots.add(spot.id, spot, expire=MAX_SPOT_AGE)
 
             self.write(safe_json_dumps("OK"))
             self.set_status(201)

server/handlers/api/alerts.py

@@ -1,6 +1,7 @@
 import copy
 import logging
 from datetime import datetime
+from queue import Queue
 from typing import Any
 
 import pytz
@@ -10,9 +11,12 @@ from tornado import httputil
 from tornado.web import Application
 
 from core.prometheus_metrics_handler import api_requests_counter
-from core.utils import safe_json_dumps
+from core.utils import safe_json_dumps, empty_queue
 from data.lookup_credentials import extract_credentials
 
+SSE_HANDLER_MAX_QUEUE_SIZE = 100
+SSE_HANDLER_QUEUE_CHECK_INTERVAL = 5000
+
 
 class APIAlertsHandler(tornado.web.RequestHandler):
     """API request handler for /api/v1/alerts"""
@@ -69,14 +73,16 @@ class APIAlertsStreamHandler(tornado_eventsource.handler.EventSourceHandler):
     """API request handler for /api/v1/alerts/stream"""
 
     def __init__(self, application, request, **kwargs: Any):
-        self._sse_alert_broadcaster = None
+        self._sse_alert_queues = None
         self._web_server_metrics = None
         self._query_params = None
         self._credentials = None
+        self._alert_queue = None
+        self._heartbeat = None
         super().__init__(application, request, **kwargs)
 
-    def initialize(self, _sse_alert_broadcaster, web_server_metrics):
+    def initialize(self, sse_alert_queues, web_server_metrics):
-        self._sse_alert_broadcaster = _sse_alert_broadcaster
+        self._sse_alert_queues = sse_alert_queues
         self._web_server_metrics = web_server_metrics
 
     def custom_headers(self):
@@ -98,32 +104,59 @@ class APIAlertsStreamHandler(tornado_eventsource.handler.EventSourceHandler):
             self._query_params = {k: v[0].decode("utf-8") for k, v in self.request.arguments.items()}
             self._credentials = extract_credentials(self._query_params)
 
+            # Create a alert queue and add it to the web server's list. The web server will fill this when alerts arrive
+            self._alert_queue = Queue(maxsize=SSE_HANDLER_MAX_QUEUE_SIZE)
+            self._sse_alert_queues.append(self._alert_queue)
+
+            # Set up a timed callback to check if anything is in the queue
+            self._heartbeat = tornado.ioloop.PeriodicCallback(self._callback, SSE_HANDLER_QUEUE_CHECK_INTERVAL)
+            self._heartbeat.start()
+
             # Flush headers immediately so nginx doesn't time out waiting for a response
             self.write_message("keepalive", "")
 
-            # Register to handle new alerts arriving. The callback() method will get called with the new alert as an
-            # argument.
-            self._sse_alert_broadcaster.register(self)
-
         except Exception as e:
             logging.warning("Exception when serving SSE socket: %s", e, exc_info=True)
             self.close()
 
     def close(self):
-        """When the user closes the socket, deregister ourselves from the alert broadcaster"""
+        """When the user closes the socket, empty our queue and remove it from the list so the server no longer fills it"""
-
-        self._sse_alert_broadcaster.unregister(self)
-        super().close()
-
-    def callback(self, alert):
-        """Callback when a new alert arrives"""
 
         try:
-            if alert_allowed_by_query(alert, self._query_params):
+            if self._alert_queue in self._sse_alert_queues:
-                if self._credentials:
+                self._sse_alert_queues.remove(self._alert_queue)
-                    alert = copy.deepcopy(alert)
+            empty_queue(self._alert_queue)
-                    alert.infer_missing(self._credentials)
+        except:
-                self.write_message(msg=safe_json_dumps(alert))
+            pass
+        try:
+            self._heartbeat.stop()
+        except:
+            pass
+        self._alert_queue = None
+        super().close()
+
+    def _callback(self):
+        """Callback to check if anything has arrived in the queue, and if so send it to the client"""
+
+        try:
+            if self._alert_queue:
+                if not self._alert_queue.empty():
+                    while not self._alert_queue.empty():
+                        alert = self._alert_queue.get()
+                        # If the new alert matches our param filters, send it to the client. If not, ignore it.
+                        if alert_allowed_by_query(alert, self._query_params):
+                            if self._credentials:
+                                alert = copy.deepcopy(alert)
+                                alert.infer_missing(self._credentials)
+                            self.write_message(msg=safe_json_dumps(alert))
+
+                else:
+                    # Send a keepalive comment if the queue was empty
+                    self.write_message("keepalive", "")
+
+                if self._alert_queue not in self._sse_alert_queues:
+                    logging.error("Web server cleared up a queue of an active connection!")
+                    self.close()
         except Exception as e:
             logging.warning("Exception in SSE callback, connection will be closed: %s", e, exc_info=True)
             self.close()
@@ -136,7 +169,7 @@ def get_alert_list_with_filters(all_alerts, query):
     # Create a shallow copy of the alert list ordered by start time, then filter the list to reduce it only to alerts
     # that match the filter parameters in the query string. Finally, apply a limit to the number of alerts returned.
     # The list of query string filters is defined in the API docs.
-    alert_ids = all_alerts.keys()
+    alert_ids = list(all_alerts.iterkeys())
     alerts = []
     for k in alert_ids:
         a = all_alerts.get(k)

server/handlers/api/dxstats.py

@@ -40,7 +40,7 @@ class APIDxStatsHandler(tornado.web.RequestHandler):
             one_hour_ago = (datetime.now(pytz.UTC) - timedelta(hours=1)).timestamp()
             counts = Counter()
 
-            for key in self._spots.keys():
+            for key in self._spots.iterkeys():
                 spot = self._spots.get(key)
                 if spot is None:
                     continue

server/handlers/api/spots.py

@@ -1,6 +1,7 @@
 import copy
 import logging
 from datetime import datetime, timedelta
+from queue import Queue
 from typing import Any
 
 import pytz
@@ -10,9 +11,12 @@ from tornado import httputil
 from tornado.web import Application
 
 from core.prometheus_metrics_handler import api_requests_counter
-from core.utils import safe_json_dumps
+from core.utils import safe_json_dumps, empty_queue
 from data.lookup_credentials import extract_credentials
 
+SSE_HANDLER_MAX_QUEUE_SIZE = 1000
+SSE_HANDLER_QUEUE_CHECK_INTERVAL = 5000
+
 
 class APISpotsHandler(tornado.web.RequestHandler):
     """API request handler for /api/v1/spots"""
@@ -69,14 +73,16 @@ class APISpotsStreamHandler(tornado_eventsource.handler.EventSourceHandler):
     """API request handler for /api/v1/spots/stream"""
 
     def __init__(self, application, request, **kwargs: Any):
-        self._sse_spot_broadcaster = None
+        self._sse_spot_queues = None
         self._web_server_metrics = None
         self._query_params = None
         self._credentials = None
+        self._spot_queue = None
+        self._heartbeat = None
         super().__init__(application, request, **kwargs)
 
-    def initialize(self, sse_spot_broadcaster, web_server_metrics):
+    def initialize(self, sse_spot_queues, web_server_metrics):
-        self._sse_spot_broadcaster = sse_spot_broadcaster
+        self._sse_spot_queues = sse_spot_queues
         self._web_server_metrics = web_server_metrics
 
     def custom_headers(self):
@@ -100,33 +106,59 @@ class APISpotsStreamHandler(tornado_eventsource.handler.EventSourceHandler):
             self._query_params = {k: v[0].decode("utf-8") for k, v in self.request.arguments.items()}
             self._credentials = extract_credentials(self._query_params)
 
+            # Create a spot queue and add it to the web server's list. The web server will fill this when spots arrive
+            self._spot_queue = Queue(maxsize=SSE_HANDLER_MAX_QUEUE_SIZE)
+            self._sse_spot_queues.append(self._spot_queue)
+
+            # Set up a timed callback to check if anything is in the queue
+            self._heartbeat = tornado.ioloop.PeriodicCallback(self._callback, SSE_HANDLER_QUEUE_CHECK_INTERVAL)
+            self._heartbeat.start()
+
             # Flush headers immediately so nginx doesn't time out waiting for a response
             self.write_message("keepalive", "")
 
-            # Register to handle new spots arriving. The callback() method will get called with the new spot as an
-            # argument.
-            self._sse_spot_broadcaster.register(self)
-
         except Exception as e:
             logging.warning("Exception when serving SSE socket: %s", e, exc_info=True)
             self.close()
 
     def close(self):
-        """When the user closes the socket, deregister ourselves from the spot broadcaster"""
+        """When the user closes the socket, empty our queue and remove it from the list so the server no longer fills it"""
-
-        self._sse_spot_broadcaster.unregister(self)
-        super().close()
-
-    def callback(self, spot):
-        """Callback when a new spot arrives"""
 
         try:
-            # If the new spot matches our param filters, send it to the client. If not, ignore it.
+            if self._spot_queue in self._sse_spot_queues:
-            if spot_allowed_by_query(spot, self._query_params):
+                self._sse_spot_queues.remove(self._spot_queue)
-                if self._credentials:
+            empty_queue(self._spot_queue)
-                    spot = copy.deepcopy(spot)
+        except:
-                    spot.infer_missing(self._credentials)
+            pass
-                self.write_message(msg=safe_json_dumps(spot))
+        try:
+            self._heartbeat.stop()
+        except:
+            pass
+        self._spot_queue = None
+        super().close()
+
+    def _callback(self):
+        """Callback to check if anything has arrived in the queue, and if so send it to the client"""
+
+        try:
+            if self._spot_queue:
+                if not self._spot_queue.empty():
+                    while not self._spot_queue.empty():
+                        spot = self._spot_queue.get()
+                        # If the new spot matches our param filters, send it to the client. If not, ignore it.
+                        if spot_allowed_by_query(spot, self._query_params):
+                            if self._credentials:
+                                spot = copy.deepcopy(spot)
+                                spot.infer_missing(self._credentials)
+                            self.write_message(msg=safe_json_dumps(spot))
+
+                else:
+                    # Send a keepalive comment if the queue was empty
+                    self.write_message("keepalive", "")
+
+                if self._spot_queue not in self._sse_spot_queues:
+                    logging.error("Web server cleared up a queue of an active connection!")
+                    self.close()
         except Exception as e:
             logging.warning("Exception in SSE callback, connection will be closed: %s", e, exc_info=True)
             self.close()
@@ -139,7 +171,7 @@ def get_spot_list_with_filters(all_spots, query):
     # Create a shallow copy of the spot list, ordered by spot time, then filter the list to reduce it only to spots
     # that match the filter parameters in the query string. Finally, apply a limit to the number of spots returned.
     # The list of query string filters is defined in the API docs.
-    spot_ids = all_spots.keys()
+    spot_ids = list(all_spots.iterkeys())
     spots = []
     for k in spot_ids:
         s = all_spots.get(k)

server/sse_broadcaster.py

@@ -1,36 +0,0 @@
-import logging
-import threading
-
-from tornado.ioloop import IOLoop
-
-
-class SSEBroadcaster:
-    """Bridge between DataStore listener callbacks (which fire on provider threads) to Tornado's async SSE handlers
-    (which live on the IOLoop thread) to avoid any interdependency between them."""
-
-    def __init__(self):
-        self._handlers = set()
-        self._lock = threading.Lock()
-        self._loop = IOLoop.current()
-
-    def register(self, handler):
-        with self._lock:
-            self._handlers.add(handler)
-
-    def unregister(self, handler):
-        with self._lock:
-            self._handlers.discard(handler)
-
-    def publish(self, value):
-        self._loop.add_callback(self._fan_out, value)
-
-    def _fan_out(self, value):
-        with self._lock:
-            handlers = list(self._handlers)
-        for handler in handlers:
-            try:
-                handler.callback(value)
-            except Exception:
-                # Connection probably dropped, ignore and de-register the handler to stop getting future items.
-                logging.debug("Failed to push to an SSE client; dropping it")
-                self.unregister(handler)

server/webserver.py

@@ -6,7 +6,7 @@ import tornado
 from tornado.web import StaticFileHandler
 
 from core.config import ALLOW_SPOTTING, WEB_SERVER_PORT, API_ONLY_MODE, LOG_WEB_REQUESTS, BASE_URL
-from core.data_store import DATA_STORE
+from core.utils import empty_queue
 from server.handlers.api.addspot import APISpotHandler
 from server.handlers.api.alerts import APIAlertsHandler, APIAlertsStreamHandler
 from server.handlers.api.dxstats import APIDxStatsHandler
@@ -18,7 +18,6 @@ from server.handlers.api.status import APIStatusHandler
 from server.handlers.manifesthandler import ManifestHandler
 from server.handlers.metrics import PrometheusMetricsHandler
 from server.handlers.pagetemplate import PageTemplateHandler
-from server.sse_broadcaster import SSEBroadcaster
 
 _HERE = os.path.dirname(__file__ or "")
 
@@ -26,12 +25,15 @@ _HERE = os.path.dirname(__file__ or "")
 class WebServer:
     """Provides the public-facing web server."""
 
-    def __init__(self):
+    def __init__(self, spots, alerts, solar_conditions, status_data):
         """Constructor"""
 
-        self._data_store = DATA_STORE
+        self._spots = spots
-        self._spot_broadcaster = SSEBroadcaster()
+        self._alerts = alerts
-        self._alert_broadcaster = SSEBroadcaster()
+        self._solar_conditions = solar_conditions
+        self._sse_spot_queues = []
+        self._sse_alert_queues = []
+        self._status_data = status_data
         self._port = WEB_SERVER_PORT
         self._api_only_mode = API_ONLY_MODE
         self._shutdown_event = asyncio.Event()
@@ -43,10 +45,6 @@ class WebServer:
             "status": "Starting"
         }
 
-        # Listen for new spots and alerts being added to the cache, so we can notify SSE clients immediately
-        DATA_STORE.spots.add_listener(self._spot_broadcaster.publish)
-        DATA_STORE.alerts.add_listener(self._alert_broadcaster.publish)
-
     def start(self):
         """Start the web server"""
 
@@ -66,20 +64,20 @@ class WebServer:
 
         # API endpoints are always enabled
         api_routes = [
-            (r"/api/v1/spots", APISpotsHandler, {"spots": self._data_store.spots, **handler_opts}),
+            (r"/api/v1/spots", APISpotsHandler, {"spots": self._spots, **handler_opts}),
-            (r"/api/v1/alerts", APIAlertsHandler, {"alerts": self._data_store.alerts, **handler_opts}),
+            (r"/api/v1/alerts", APIAlertsHandler, {"alerts": self._alerts, **handler_opts}),
             (r"/api/v1/spots/stream", APISpotsStreamHandler,
-             {"sse_spot_broadcaster": self._spot_broadcaster, **handler_opts}),
+             {"sse_spot_queues": self._sse_spot_queues, **handler_opts}),
             (r"/api/v1/alerts/stream", APIAlertsStreamHandler,
-             {"sse_alert_broadcaster": self._alert_broadcaster, **handler_opts}),
+             {"sse_alert_queues": self._sse_alert_queues, **handler_opts}),
-            (r"/api/v1/solar", APISolarConditionsHandler, {"solar_conditions": self._data_store.solar, **handler_opts}),
+            (r"/api/v1/solar", APISolarConditionsHandler, {"solar_conditions": self._solar_conditions, **handler_opts}),
-            (r"/api/v1/dxstats", APIDxStatsHandler, {"spots": self._data_store.spots, **handler_opts}),
+            (r"/api/v1/dxstats", APIDxStatsHandler, {"spots": self._spots, **handler_opts}),
-            (r"/api/v1/options", APIOptionsHandler, {"status_data": self._data_store.status_data, **handler_opts}),
+            (r"/api/v1/options", APIOptionsHandler, {"status_data": self._status_data, **handler_opts}),
-            (r"/api/v1/status", APIStatusHandler, {"status_data": self._data_store.status_data, **handler_opts}),
+            (r"/api/v1/status", APIStatusHandler, {"status_data": self._status_data, **handler_opts}),
             (r"/api/v1/lookup/call", APILookupCallHandler, {**handler_opts}),
             (r"/api/v1/lookup/sigref", APILookupSIGRefHandler, {**handler_opts}),
             (r"/api/v1/lookup/grid", APILookupGridHandler, {**handler_opts}),
-            (r"/api/v1/spot", APISpotHandler, {"spots": self._data_store.spots, **handler_opts}),
+            (r"/api/v1/spot", APISpotHandler, {"spots": self._spots, **handler_opts}),
         ]
 
         # If in API-only mode, serve a basic homepage; in normal mode, serve the usual UI routes
@@ -123,6 +121,55 @@ class WebServer:
         logging.info("You can access your copy of Spothole at " + BASE_URL)
         await self._shutdown_event.wait()
 
+    def notify_new_spot(self, spot):
+        """Internal method called when a new spot is added to the system. This is used to ping any SSE clients that are
+        awaiting a server-sent message with new spots."""
+
+        for queue in self._sse_spot_queues:
+            try:
+                queue.put(spot)
+            except:
+                # Cleanup thread was probably deleting the queue, that's fine
+                pass
+        pass
+
+    def notify_new_alert(self, alert):
+        """Internal method called when a new alert is added to the system. This is used to ping any SSE clients that are
+        awaiting a server-sent message with new spots."""
+
+        for queue in self._sse_alert_queues:
+            try:
+                queue.put(alert)
+            except:
+                # Cleanup thread was probably deleting the queue, that's fine
+                pass
+        pass
+
+    def clean_up_sse_queues(self):
+        """Clean up any SSE queues that are growing too large; probably their client disconnected and we didn't catch it
+        properly for some reason."""
+
+        for q in self._sse_spot_queues:
+            try:
+                if q.full():
+                    logging.warning(
+                        "A full SSE spot queue was found, presumably because the client disconnected strangely. It has been removed.")
+                    self._sse_spot_queues.remove(q)
+                    empty_queue(q)
+            except:
+                # Probably got deleted already on another thread
+                pass
+        for q in self._sse_alert_queues:
+            try:
+                if q.full():
+                    logging.warning(
+                        "A full SSE alert queue was found, presumably because the client disconnected strangely. It has been removed.")
+                    self._sse_alert_queues.remove(q)
+                    empty_queue(q)
+            except:
+                # Probably got deleted already on another thread
+                pass
+        pass
 
 def request_log(handler):
     """Custom log function to provide more data about requests when enabled, and to provide the ability to turn off

sigrefdataproviders/sig_ref_data_provider.py

@@ -1,28 +0,0 @@
-from datetime import datetime
-
-import pytz
-
-
-class SIGRefDataProvider:
-    """Generic SIG reference data provider class. Subclasses of this query the individual URLs or files for data."""
-
-    def __init__(self, name, provider_config):
-        """Constructor"""
-
-        self.name = name
-        self.enabled = provider_config["enabled"]
-        self.last_update_time = datetime.min.replace(tzinfo=pytz.UTC)
-        self.last_spot_time = datetime.min.replace(tzinfo=pytz.UTC)
-        self.status = "Not Started" if self.enabled else "Disabled"
-
-
-    def start(self):
-        """Start the provider. This should return immediately after spawning threads to access the remote resources"""
-
-        raise NotImplementedError("Subclasses must implement this method")
-
-
-    def stop(self):
-        """Stop any threads and prepare for application shutdown"""
-
-        raise NotImplementedError("Subclasses must implement this method")

solarconditionsproviders/giroionosonde.py

@@ -35,18 +35,6 @@ class GIROIonosonde(SolarConditionsProvider):
         self._thread = None
         self._stop_event = Event()
 
-        # Pre-populate ionosonde_data with known station names for stations not already present,
-        # so the station dropdown is available before the first poll. Does not overwrite existing
-        # entries so KC2G cache data is preserved.
-        existing = self._solar_conditions.ionosonde_data or {}
-        new_entries = {
-            s["ursi"]: {"ursi": s["ursi"], "name": s["name"], "fof2": None, "muf": None,
-                        "luf": None, "band_states": None}
-            for s in self._stations if s["ursi"] not in existing
-        }
-        if new_entries:
-            self.update_data({"ionosonde_data": {**existing, **new_entries}})
-
     @staticmethod
     def _load_stations():
         stations = []
@@ -56,6 +44,21 @@ class GIROIonosonde(SolarConditionsProvider):
                     stations.append({"ursi": row[0].strip(), "name": row[1].strip()})
         return stations
 
+    def setup(self, solar_conditions, solar_conditions_cache):
+        """Pre-populate ionosonde_data with known station names for stations not already present,
+        so the station dropdown is available before the first poll. Does not overwrite existing
+        entries so KC2G cache data is preserved."""
+
+        super().setup(solar_conditions, solar_conditions_cache)
+        existing = solar_conditions.ionosonde_data or {}
+        new_entries = {
+            s["ursi"]: {"ursi": s["ursi"], "name": s["name"], "fof2": None, "muf": None,
+                        "luf": None, "band_states": None}
+            for s in self._stations if s["ursi"] not in existing
+        }
+        if new_entries:
+            self.update_data({"ionosonde_data": {**existing, **new_entries}})
+
     def start(self):
         logging.info(f"Set up query of GIRO ionosonde data API every {POLL_INTERVAL} seconds.")
         self._thread = Thread(target=self._run, daemon=True)

solarconditionsproviders/solar_conditions_provider.py

@@ -2,8 +2,6 @@ from datetime import datetime
 
 import pytz
 
-from core.data_store import DATA_STORE
-
 
 class SolarConditionsProvider:
     """Generic solar conditions provider class. Subclasses of this query individual APIs for space weather and
@@ -12,11 +10,18 @@ class SolarConditionsProvider:
     def __init__(self, name, provider_config):
         """Constructor"""
 
+        self._solar_conditions_cache = None
         self.name = name
         self.enabled = provider_config["enabled"]
         self.last_update_time = datetime.min.replace(tzinfo=pytz.UTC)
         self.status = "Not Started" if self.enabled else "Disabled"
-        self._solar_conditions = DATA_STORE.solar_conditions
+        self._solar_conditions = None
+
+    def setup(self, solar_conditions, solar_conditions_cache):
+        """Set up the provider, giving it the solar conditions object and its backing cache"""
+
+        self._solar_conditions = solar_conditions
+        self._solar_conditions_cache = solar_conditions_cache
 
     def start(self):
         """Start the provider. This should return immediately after spawning threads to access the remote resources"""
@@ -36,3 +41,4 @@ class SolarConditionsProvider:
                 if hasattr(self._solar_conditions, key):
                     setattr(self._solar_conditions, key, value)
             self._solar_conditions.infer_descriptions()
+            self._solar_conditions_cache['solar_conditions'] = self._solar_conditions

spothole.py

@@ -5,15 +5,23 @@ import os
 import signal
 import sys
 
+from diskcache import Cache
+
+from core.cleanup import CleanupTimer
 from core.config import config, SERVER_OWNER_CALLSIGN, LOG_LEVEL
 from core.constants import SOFTWARE_VERSION
-from core.data_store import DATA_STORE
 from core.lookup_helper import lookup_helper
 from core.status_reporter import StatusReporter
+from data.solar_conditions import SolarConditions
 from server.webserver import WebServer
 
 # Globals
+spots = Cache('cache/spots_cache')
+alerts = Cache('cache/alerts_cache')
+solar_conditions_cache = Cache('cache/solar_conditions_cache')
+solar_conditions = solar_conditions_cache.get('solar_conditions', SolarConditions())
 web_server = None
+status_data = {}
 spot_providers = []
 alert_providers = []
 solar_condition_providers = []
@@ -38,7 +46,13 @@ def shutdown(_signum=None, _frame=None):
     for scp in solar_condition_providers:
         if scp.enabled:
             scp.stop()
-    DATA_STORE.close()
+    if cleanup_timer:
+        cleanup_timer.stop()
+    if lookup_helper:
+        lookup_helper.stop()
+    spots.close()
+    alerts.close()
+    solar_conditions_cache.close()
     os._exit(0)
 
 
@@ -89,12 +103,13 @@ if __name__ == '__main__':
     lookup_helper.start()
 
     # Set up web server
-    web_server = WebServer()
+    web_server = WebServer(spots=spots, alerts=alerts, solar_conditions=solar_conditions, status_data=status_data)
 
     # Fetch, set up and start spot providers
     for entry in config["spot-providers"]:
         spot_providers.append(get_spot_provider_from_config(entry))
     for p in spot_providers:
+        p.setup(spots=spots, web_server=web_server)
         if p.enabled:
             p.start()
 
@@ -102,6 +117,7 @@ if __name__ == '__main__':
     for entry in config["alert-providers"]:
         alert_providers.append(get_alert_provider_from_config(entry))
     for p in alert_providers:
+        p.setup(alerts=alerts, web_server=web_server)
         if p.enabled:
             p.start()
 
@@ -109,11 +125,17 @@ if __name__ == '__main__':
     for entry in config.get("solar-condition-providers", []):
         solar_condition_providers.append(get_solar_conditions_provider_from_config(entry))
     for p in solar_condition_providers:
+        p.setup(solar_conditions=solar_conditions, solar_conditions_cache=solar_conditions_cache)
         if p.enabled:
             p.start()
 
+    # Set up timer to clear spot list of old data
+    cleanup_timer = CleanupTimer(spots=spots, alerts=alerts, web_server=web_server, cleanup_interval=60)
+    cleanup_timer.start()
+
     # Set up status reporter
-    status_reporter = StatusReporter(web_server=web_server, spot_providers=spot_providers,
+    status_reporter = StatusReporter(status_data=status_data, spots=spots, alerts=alerts, web_server=web_server,
+                                     cleanup_timer=cleanup_timer, spot_providers=spot_providers,
                                      alert_providers=alert_providers,
                                      solar_condition_providers=solar_condition_providers, run_interval=5)
     status_reporter.start()

spotproviders/gma.py

@@ -3,8 +3,8 @@ from datetime import datetime
 
 import pytz
 
+from core.cache_utils import SEMI_STATIC_URL_DATA_CACHE
 from core.constants import HTTP_HEADERS
-from core.url_data_cache import URL_DATA_CACHE
 from data.sig_ref import SIGRef
 from data.spot import Spot
 from spotproviders.http_spot_provider import HTTPSpotProvider
@@ -34,6 +34,10 @@ class GMA(HTTPSpotProvider):
         if "RCD" in http_response.json():
             for source_spot in http_response.json()["RCD"]:
                 # Convert to our spot format
+                # Seen GMA spots with no (or empty) lat/lon
+                lat = float(source_spot["LAT"]) if (source_spot["LAT"] and source_spot["LAT"] != "") else None
+                lon = float(source_spot["LON"]) if (source_spot["LON"] and source_spot["LON"] != "") else None
+
                 spot = Spot(source=self.name,
                             dx_call=source_spot["ACTIVATOR"].upper(),
                             de_call=source_spot["SPOTTER"].upper(),
@@ -43,21 +47,19 @@ class GMA(HTTPSpotProvider):
                             # Filter out some weird mode strings
                             mode=source_spot["MODE"].upper() if "<>" not in source_spot["MODE"] else None,
                             comment=source_spot["TEXT"],
-                            sig_refs=[SIGRef(id=source_spot["REF"], sig="", name=source_spot["NAME"])],
+                            sig_refs=[SIGRef(id=source_spot["REF"], sig="", name=source_spot["NAME"], latitude=lat,
+                                             longitude=lon)],
                             time=datetime.strptime(source_spot["DATE"] + source_spot["TIME"], "%Y%m%d%H%M").replace(
                                 tzinfo=pytz.UTC).timestamp(),
-                            # Seen GMA spots with no (or empty) lat/lon
+                            dx_latitude=lat,
-                            dx_latitude=float(source_spot["LAT"]) if (
+                            dx_longitude=lon,
-                                    source_spot["LAT"] and source_spot["LAT"] != "") else None,
-                            dx_longitude=float(source_spot["LON"]) if (
-                                    source_spot["LON"] and source_spot["LON"] != "") else None,
                             qrt=source_spot["QRG"] == "QRT")
 
                 # GMA doesn't give what programme (SIG) the reference is for until we separately look it up.
                 if "REF" in source_spot:
                     try:
-                        ref_response = URL_DATA_CACHE.get(self.REF_INFO_URL_ROOT + source_spot["REF"],
+                        ref_response = SEMI_STATIC_URL_DATA_CACHE.get(self.REF_INFO_URL_ROOT + source_spot["REF"],
-                                                          headers=HTTP_HEADERS)
+                                                                      headers=HTTP_HEADERS)
                         # Sometimes this is blank even if it's a 200 response, so handle that
                         if ref_response.ok and ref_response.text is not None and ref_response.text != "":
                             ref_info = ref_response.json()

spotproviders/hema.py

@@ -59,7 +59,8 @@ class HEMA(HTTPSpotProvider):
                                     mode=freq_mode_match.group(2).upper(),
                                     comment=spotter_comment_match.group(2),
                                     sig="HEMA",
-                                    sig_refs=[SIGRef(id=spot_items[3].upper(), sig="HEMA", name=spot_items[4])],
+                                    sig_refs=[SIGRef(id=spot_items[3].upper(), sig="HEMA", name=spot_items[4],
+                                                     latitude=float(spot_items[7]), longitude=float(spot_items[8]))],
                                     time=datetime.strptime(spot_items[0], "%d/%m/%Y %H:%M").replace(
                                         tzinfo=pytz.UTC).timestamp(),
                                     dx_latitude=float(spot_items[7]),

spotproviders/pota.py

@@ -29,7 +29,8 @@ class POTA(HTTPSpotProvider):
                         mode=source_spot["mode"].upper(),
                         comment=source_spot["comments"],
                         sig="POTA",
-                        sig_refs=[SIGRef(id=source_spot["reference"], sig="POTA", name=source_spot["name"])],
+                        sig_refs=[SIGRef(id=source_spot["reference"], sig="POTA", name=source_spot["name"],
+                                         latitude=source_spot["latitude"], longitude=source_spot["longitude"])],
                         time=datetime.strptime(source_spot["spotTime"], "%Y-%m-%dT%H:%M:%S").replace(
                             tzinfo=pytz.UTC).timestamp(),
                         dx_grid=source_spot["grid6"],

spotproviders/sota.py

@@ -19,8 +19,6 @@ class SOTA(HTTPSpotProvider):
     # The actual data lookup all happens after parsing and checking the epoch.
     EPOCH_URL = "https://api-db2.sota.org.uk/api/spots/epoch"
     SPOTS_URL = "https://api-db2.sota.org.uk/api/spots/60/all/all"
-    # SOTA spots don't contain lat/lon, we need a separate lookup for that
-    SUMMIT_URL_ROOT = "https://api-db2.sota.org.uk/api/summits/"
 
     def __init__(self, provider_config):
         super().__init__("SOTA", provider_config, self.EPOCH_URL, self.POLL_INTERVAL_SEC)
@@ -51,8 +49,12 @@ class SOTA(HTTPSpotProvider):
                                 mode=source_spot["mode"].upper(),
                                 comment=source_spot["comments"],
                                 sig="SOTA",
-                                sig_refs=[SIGRef(id=source_spot["summitCode"], sig="SOTA", name=source_spot["summitName"],
+                                sig_refs=[SIGRef(id=source_spot["summitCode"], sig="SOTA",
+                                                 name=source_spot["summitName"], latitude=source_spot["latitude"],
+                                                 longitude=source_spot["longitude"],
                                                  activation_score=source_spot["points"])],
+                                dx_latitude=source_spot["latitude"],
+                                dx_longitude=source_spot["longitude"],
                                 time=datetime.fromisoformat(source_spot["timeStamp"].replace("Z", "+00:00")).timestamp())
 
                     # Add to our list. Don't worry about de-duping, removing old spots etc. at this point; other code will do

spotproviders/spot_provider.py

@@ -2,7 +2,7 @@ from datetime import datetime
 
 import pytz
 
-from core.data_store import DATA_STORE
+from core.config import MAX_SPOT_AGE
 
 
 class SpotProvider:
@@ -16,7 +16,14 @@ class SpotProvider:
         self.last_update_time = datetime.min.replace(tzinfo=pytz.UTC)
         self.last_spot_time = datetime.min.replace(tzinfo=pytz.UTC)
         self.status = "Not Started" if self.enabled else "Disabled"
-        self._spots = DATA_STORE.spots
+        self._spots = None
+        self._web_server = None
+
+    def setup(self, spots, web_server):
+        """Set up the provider, e.g. giving it the spot list to work from"""
+
+        self._spots = spots
+        self._web_server = web_server
 
     def start(self):
         """Start the provider. This should return immediately after spawning threads to access the remote resources"""
@@ -52,7 +59,10 @@ class SpotProvider:
 
     def _add_spot(self, spot):
         if not spot.expired():
-            self._spots.set(spot.id, spot)
+            self._spots.add(spot.id, spot, expire=MAX_SPOT_AGE)
+            # Ping the web server in case we have any SSE connections that need to see this immediately
+            if self._web_server:
+                self._web_server.notify_new_spot(spot)
 
     def stop(self):
         """Stop any threads and prepare for application shutdown"""

spotproviders/tiles.py

@@ -31,7 +31,8 @@ class Tiles(HTTPSpotProvider):
                         # Tiles spots can include POTA & SOTA references, but ignore those on the basis that we will get them separately from the POTA/SOTA providers anyway.
                         # Just take the grid reference itself as the single Tiles SIG reference.
                         sig_refs=[SIGRef(id=source_spot["maidenhead_grid"], sig="Tiles",
-                                         name=source_spot["maidenhead_grid"])],
+                                         name=source_spot["maidenhead_grid"], latitude=source_spot["latitude"],
+                                         longitude=source_spot["longitude"])],
                         time=datetime.fromisoformat(source_spot["created_at"].replace("Z", "+00:00")).timestamp(),
                         dx_grid=source_spot["maidenhead_grid"],
                         dx_latitude=source_spot["latitude"],

spotproviders/wwbota.py

@@ -20,7 +20,8 @@ class WWBOTA(SSESpotProvider):
         # n-fer activations.
         refs = []
         for ref in source_spot["references"]:
-            sigref = SIGRef(id=ref["reference"], sig="WWBOTA", name=ref["name"])
+            sigref = SIGRef(id=ref["reference"], sig="WWBOTA", name=ref["name"], latitude=ref["lat"],
+                            longitude=ref["long"])
             refs.append(sigref)
 
         spot = Spot(source=self.name,

spotproviders/wwff.py

@@ -29,7 +29,8 @@ class WWFF(HTTPSpotProvider):
                         mode=source_spot["mode"].upper(),
                         comment=source_spot["remarks"],
                         sig="WWFF",
-                        sig_refs=[SIGRef(id=source_spot["reference"], sig="WWFF", name=source_spot["reference_name"])],
+                        sig_refs=[SIGRef(id=source_spot["reference"], sig="WWFF", name=source_spot["reference_name"],
+                                         latitude=source_spot["latitude"], longitude=source_spot["longitude"])],
                         time=datetime.fromtimestamp(source_spot["spot_time"], tz=pytz.UTC).timestamp(),
                         dx_latitude=source_spot["latitude"],
                         dx_longitude=source_spot["longitude"])

static/vendor/js/leaflet-workedallbritainireland.js

@@ -16,7 +16,7 @@ L.WorkedAllBritainIreland = L.LayerGroup.extend({
 
         // Workaround to load the geodesy modules in non-modular code. Once we have loaded all three modules, trigger a
         // first draw.
-        import("https://misc.ianrenton.com/Leaflet.WorkedAllBritainIreland/modules/geodesy/osgridref.js")
+        import(new URL('./modules/geodesy/osgridref.js', import.meta.url).href)
             .then(module => {
                 this._osGridLibrary = module;
                 if (this._ieGridLibrary && this._utmLibrary) {
@@ -27,7 +27,7 @@ L.WorkedAllBritainIreland = L.LayerGroup.extend({
                 console.log("Error loading OS Grid Ref library, GB WAB squares may not be available.");
                 console.log(error);
             });
-        import("https://misc.ianrenton.com/Leaflet.WorkedAllBritainIreland/modules/geodesy/iegridref.js")
+        import(new URL('./modules/geodesy/iegridref.js', import.meta.url).href)
             .then(module => {
                 this._ieGridLibrary = module;
                 if (this._osGridLibrary && this._utmLibrary) {
@@ -38,7 +38,7 @@ L.WorkedAllBritainIreland = L.LayerGroup.extend({
                 console.log("Error loading IE Grid Ref library, NI WAB squares may not be available.");
                 console.log(error);
             });
-        import("https://misc.ianrenton.com/Leaflet.WorkedAllBritainIreland/modules/geodesy/utm_ci.js")
+        import(new URL('./modules/geodesy/utm_ci.js', import.meta.url).href)
             .then(module => {
                 this._utmLibrary = module;
                 if (this._osGridLibrary && this._ieGridLibrary) {

static/vendor/js/modules/geodesy/LICENSE

@@ -0,0 +1,22 @@
+The MIT License (MIT)
+
+Copyright (c) 2014 Chris Veness
+With some additional code & modifications by Ian Renton, 2025
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

static/vendor/js/modules/geodesy/iegridref.js

@@ -0,0 +1,326 @@
+/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  */
+/* Ordnance Survey of Ireland Grid Reference funcs   (c) Chris Veness 2005-2021 & Ian Renton 2025 */
+/*                                                                                   MIT Licence  */
+/* www.movable-type.co.uk/scripts/latlong-gridref.html                                            */
+/* www.movable-type.co.uk/scripts/geodesy-library.html#IeGridRef                                  */
+/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  */
+
+import LatLonEllipsoidal, { Dms } from 'https://cdn.jsdelivr.net/npm/geodesy@2/latlon-ellipsoidal-datum.js';
+
+
+/**
+ * Ordnance Survey of Ireland & Northern Ireland grid reference calculations, based on the
+ * IeGridRef class in the geodesy library at https://github.com/chrisveness/geodesy
+ */
+
+/* IeGridRef  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
+
+
+const nationalGrid = {
+    trueOrigin:  { lat: 53.5, lon: -8 },                 // true origin of Irish grid 53°30′N, 8°W
+    falseOrigin: { easting: -200e3, northing: -250e3 }, // easting & northing of false origin, metres from true origin
+    scaleFactor: 1.000035,                         // scale factor on central meridian
+    ellipsoid:   LatLonEllipsoidal.ellipsoids.Airy1830,
+};
+
+/**
+ * Irish Grid References with methods to parse and convert them to latitude/longitude points.
+ */
+class IeGridRef {
+
+    /**
+     * Creates an IeGridRef object.
+     *
+     * @param {number} easting - Easting in metres from OS Grid false origin.
+     * @param {number} northing - Northing in metres from OS Grid false origin.
+     *
+     * @example
+     *   import IeGridRef from '/js/geodesy/IeGridRef.js';
+     *   const gridref = new IeGridRef(651409, 313177);
+     */
+    constructor(easting, northing) {
+        this.easting = Number(easting);
+        this.northing = Number(northing);
+
+        if (isNaN(easting)  || this.easting<0  || this.easting>7000e3) throw new RangeError(`invalid easting ‘${easting}’`);
+        if (isNaN(northing) || this.northing<0 || this.northing>13000e3) throw new RangeError(`invalid northing ‘${northing}’`);
+    }
+
+
+    /**
+     * Converts ‘this’ Irish Grid Reference easting/northing coordinate to latitude/longitude
+     * (SW corner of grid square).
+     *
+     * While OS Grid References are based on OSGB-36, the Ordnance Survey have deprecated the use of
+     * OSGB-36 for latitude/longitude coordinates (in favour of WGS-84), hence this function returns
+     * WGS-84 by default, with OSGB-36 as an option. See www.ordnancesurvey.co.uk/blog/2014/12/2.
+     *
+     * Note formulation implemented here due to Thomas, Redfearn, etc is as published by OS, but is
+     * inferior to Krüger as used by e.g. Karney 2011.
+     *
+     * @param   {LatLon.datum} [datum=WGS84] - Datum to convert grid reference into.
+     * @returns {LatLon}       Latitude/longitude of supplied grid reference.
+     *
+     * @example
+     *   const gridref = new IeGridRef(651409.903, 313177.270);
+     *   const pWgs84 = gridref.toLatLon();                    // 52°39′28.723″N, 001°42′57.787″E
+     *   // to obtain (historical) OSGB36 lat/lon point:
+     *   const pOsgb = gridref.toLatLon(LatLon.datums.OSGB36); // 52°39′27.253″N, 001°43′04.518″E
+     */
+    toLatLon(datum=LatLonEllipsoidal.datums.WGS84) {
+        const { easting: E, northing: N } = this;
+
+        const { a, b } = nationalGrid.ellipsoid;            // a = 6377563.396, b = 6356256.909
+        const φ0 = nationalGrid.trueOrigin.lat.toRadians(); // latitude of true origin
+        const λ0 = nationalGrid.trueOrigin.lon.toRadians(); // longitude of true origin
+        const E0 = -nationalGrid.falseOrigin.easting;       // easting of true origin
+        const N0 = -nationalGrid.falseOrigin.northing;      // northing of true origin
+        const F0 = nationalGrid.scaleFactor;                // scale factor
+
+        const e2 = 1 - (b*b)/(a*a);                         // eccentricity squared
+        const n = (a-b)/(a+b), n2 = n*n, n3 = n*n*n;        // n, n², n³
+
+        let φ=φ0, M=0;
+        do {
+            φ = (N-N0-M)/(a*F0) + φ;
+
+            const Ma = (1 + n + (5/4)*n2 + (5/4)*n3) * (φ-φ0);
+            const Mb = (3*n + 3*n2 + (21/8)*n3) * Math.sin(φ-φ0) * Math.cos(φ+φ0);
+            const Mc = ((15/8)*n2 + (15/8)*n3) * Math.sin(2*(φ-φ0)) * Math.cos(2*(φ+φ0));
+            const Md = (35/24)*n3 * Math.sin(3*(φ-φ0)) * Math.cos(3*(φ+φ0));
+            M = b * F0 * (Ma - Mb + Mc - Md);                // meridional arc
+
+        } while (Math.abs(N-N0-M) >= 0.00001);  // ie until < 0.01mm
+
+        const cosφ = Math.cos(φ), sinφ = Math.sin(φ);
+        const ν = a*F0/Math.sqrt(1-e2*sinφ*sinφ);            // nu = transverse radius of curvature
+        const ρ = a*F0*(1-e2)/Math.pow(1-e2*sinφ*sinφ, 1.5); // rho = meridional radius of curvature
+        const η2 = ν/ρ-1;                                    // eta = ?
+
+        const tanφ = Math.tan(φ);
+        const tan2φ = tanφ*tanφ, tan4φ = tan2φ*tan2φ, tan6φ = tan4φ*tan2φ;
+        const secφ = 1/cosφ;
+        const ν3 = ν*ν*ν, ν5 = ν3*ν*ν, ν7 = ν5*ν*ν;
+        const VII = tanφ/(2*ρ*ν);
+        const VIII = tanφ/(24*ρ*ν3)*(5+3*tan2φ+η2-9*tan2φ*η2);
+        const IX = tanφ/(720*ρ*ν5)*(61+90*tan2φ+45*tan4φ);
+        const X = secφ/ν;
+        const XI = secφ/(6*ν3)*(ν/ρ+2*tan2φ);
+        const XII = secφ/(120*ν5)*(5+28*tan2φ+24*tan4φ);
+        const XIIA = secφ/(5040*ν7)*(61+662*tan2φ+1320*tan4φ+720*tan6φ);
+
+        const dE = (E-E0), dE2 = dE*dE, dE3 = dE2*dE, dE4 = dE2*dE2, dE5 = dE3*dE2, dE6 = dE4*dE2, dE7 = dE5*dE2;
+        φ = φ - VII*dE2 + VIII*dE4 - IX*dE6;
+        const λ = λ0 + X*dE - XI*dE3 + XII*dE5 - XIIA*dE7;
+
+        let point = new LatLon_IeGridRef(φ.toDegrees(), λ.toDegrees(), 0, LatLonEllipsoidal.datums.OSGB36);
+
+        if (datum != LatLonEllipsoidal.datums.OSGB36) {
+            // if point is required in datum other than OSGB36, convert it
+            point = point.convertDatum(datum);
+            // convertDatum() gives us a LatLon: convert to LatLon_IeGridRef which includes toOsGrid()
+            point = new LatLon_IeGridRef(point.lat, point.lon, point.height, point.datum);
+        }
+
+        return point;
+    }
+
+
+    /**
+     * Parses grid reference to IeGridRef object.
+     *
+     * Accepts standard grid references (eg 'G 387 148'), with or without whitespace separators, from
+     * two-digit references up to 10-digit references (1m × 1m square), or fully numeric comma-separated
+     * references in metres (eg '438700,114800').
+     *
+     * @param   {string}    gridref - Standard format OS Grid Reference.
+     * @returns {IeGridRef} Numeric version of grid reference in metres from false origin (SW corner of
+     *   supplied grid square).
+     * @throws  {Error}     Invalid grid reference.
+     *
+     * @example
+     *   const grid = IeGridRef.parse('G 51409 13177'); // grid: { easting: 651409, northing: 313177 }
+     */
+    static parse(gridref) {
+        gridref = String(gridref).trim();
+
+        // check for fully numeric comma-separated gridref format
+        let match = gridref.match(/^(\d+),\s*(\d+)$/);
+        if (match) return new IeGridRef(match[1], match[2]);
+
+        // validate format
+        match = gridref.match(/^[ABCDEFGHJKLMNOPQRSTUVWXYZ]\s*[0-9]+\s*[0-9]+$/i);
+        if (!match) throw new Error(`invalid grid reference ‘${gridref}’`);
+
+        // get numeric values of letter references, mapping A->0, B->1, C->2, etc:
+        let l1 = gridref.toUpperCase().charCodeAt(0) - 'A'.charCodeAt(0); // 100km square
+        // shuffle down letters after 'I' since 'I' is not used in grid:
+        if (l1 > 7) l1--;
+
+        // convert grid letters into 100km-square indexes from false origin (grid square SV):
+        const e100km = l1 % 5;
+        const n100km = 4 - Math.floor(l1 / 5);
+
+        // skip grid letters to get numeric (easting/northing) part of ref
+        let en = gridref.slice(1).trim().split(/\s+/);
+        // if e/n not whitespace separated, split half way
+        if (en.length == 1) en = [ en[0].slice(0, en[0].length / 2), en[0].slice(en[0].length / 2) ];
+
+        // validation
+        if (en[0].length != en[1].length) throw new Error(`invalid grid reference ‘${gridref}’`);
+
+        // standardise to 10-digit refs (metres)
+        en[0] = en[0].padEnd(5, '0');
+        en[1] = en[1].padEnd(5, '0');
+
+        const e = e100km + en[0];
+        const n = n100km + en[1];
+
+        return new IeGridRef(e, n);
+    }
+
+
+    /**
+     * Converts ‘this’ numeric grid reference to standard OS of Ireland Grid Reference.
+     *
+     * @param   {number} [digits=10] - Precision of returned grid reference (10 digits = metres);
+     *   digits=0 will return grid reference in numeric format.
+     * @returns {string} This grid reference in standard format.
+     *
+     * @example
+     *   const gridref = new IeGridRef(651409, 313177).toString(8); // 'TG 5140 1317'
+     *   const gridref = new IeGridRef(651409, 313177).toString(0); // '651409,313177'
+     */
+    toString(digits=10) {
+        if (![ 0,2,4,6,8,10,12,14,16 ].includes(Number(digits))) throw new RangeError(`invalid precision ‘${digits}’`); // eslint-disable-line comma-spacing
+
+        let { easting: e, northing: n } = this;
+
+        // use digits = 0 to return numeric format (in metres) - note northing may be >= 1e7
+        if (digits == 0) {
+            const format = { useGrouping: false,  minimumIntegerDigits: 6, maximumFractionDigits: 3 };
+            const ePad = e.toLocaleString('en', format);
+            const nPad = n.toLocaleString('en', format);
+            return `${ePad},${nPad}`;
+        }
+
+        // get the 100km-grid indices
+        const e100km = Math.floor(e / 100000), n100km = Math.floor(n / 100000);
+
+        // translate those into the numeric equivalent of the grid letters
+        let l1 = (n100km) * 5 % 25 + e100km % 5;
+        return null; // haven't done this maths yet
+
+        // compensate for skipped 'I' and calculate grid letter
+        if (l1 > 7) l1++;
+        const letter = String.fromCharCode(l1 + 'A'.charCodeAt(0));
+
+        // strip 100km-grid indices from easting & northing, and reduce precision
+        e = Math.floor((e % 100000) / Math.pow(10, 5 - digits / 2));
+        n = Math.floor((n % 100000) / Math.pow(10, 5 - digits / 2));
+
+        // pad eastings & northings with leading zeros
+        e = e.toString().padStart(digits/2, '0');
+        n = n.toString().padStart(digits/2, '0');
+
+        return `${letter} ${e} ${n}`;
+    }
+
+}
+
+
+/* LatLon_IeGridRef - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
+
+
+/**
+ * Extends LatLon class with method to convert LatLon point to Irish Grid Reference.
+ *
+ * @extends LatLonEllipsoidal
+ */
+class LatLon_IeGridRef extends LatLonEllipsoidal {
+
+    /**
+     * Converts latitude/longitude to Ordnance Survey of Ireland grid reference easting/northing coordinate.
+     *
+     * @returns {IeGridRef} Irish Grid Reference easting/northing.
+     *
+     * @example
+     *   const grid = new LatLon(52.65798, 1.71605).toOsGrid(); // TG 51409 13177
+     *   // for conversion of (historical) OSGB36 latitude/longitude point:
+     *   const grid = new LatLon(52.65798, 1.71605).toOsGrid(LatLon.datums.OSGB36);
+     */
+    toOsGrid() {
+        // if necessary convert to OSGB36 first
+        const point = this.datum == LatLonEllipsoidal.datums.OSGB36
+            ? this
+            : this.convertDatum(LatLonEllipsoidal.datums.OSGB36);
+
+        const φ = point.lat.toRadians();
+        const λ = point.lon.toRadians();
+
+        const { a, b } = nationalGrid.ellipsoid;            // a = 6377563.396, b = 6356256.909
+        const φ0 = nationalGrid.trueOrigin.lat.toRadians(); // latitude of true origin
+        const λ0 = nationalGrid.trueOrigin.lon.toRadians(); // longitude of true origin
+        const E0 = -nationalGrid.falseOrigin.easting;       // easting of true origin
+        const N0 = -nationalGrid.falseOrigin.northing;      // northing of true origin
+        const F0 = nationalGrid.scaleFactor;                // scale factor
+
+        const e2 = 1 - (b*b)/(a*a);                          // eccentricity squared
+        const n = (a-b)/(a+b), n2 = n*n, n3 = n*n*n;         // n, n², n³
+
+        const cosφ = Math.cos(φ), sinφ = Math.sin(φ);
+        const ν = a*F0/Math.sqrt(1-e2*sinφ*sinφ);            // nu = transverse radius of curvature
+        const ρ = a*F0*(1-e2)/Math.pow(1-e2*sinφ*sinφ, 1.5); // rho = meridional radius of curvature
+        const η2 = ν/ρ-1;                                    // eta = ?
+
+        const Ma = (1 + n + (5/4)*n2 + (5/4)*n3) * (φ-φ0);
+        const Mb = (3*n + 3*n2 + (21/8)*n3) * Math.sin(φ-φ0) * Math.cos(φ+φ0);
+        const Mc = ((15/8)*n2 + (15/8)*n3) * Math.sin(2*(φ-φ0)) * Math.cos(2*(φ+φ0));
+        const Md = (35/24)*n3 * Math.sin(3*(φ-φ0)) * Math.cos(3*(φ+φ0));
+        const M = b * F0 * (Ma - Mb + Mc - Md);              // meridional arc
+
+        const cos3φ = cosφ*cosφ*cosφ;
+        const cos5φ = cos3φ*cosφ*cosφ;
+        const tan2φ = Math.tan(φ)*Math.tan(φ);
+        const tan4φ = tan2φ*tan2φ;
+
+        const I = M + N0;
+        const II = (ν/2)*sinφ*cosφ;
+        const III = (ν/24)*sinφ*cos3φ*(5-tan2φ+9*η2);
+        const IIIA = (ν/720)*sinφ*cos5φ*(61-58*tan2φ+tan4φ);
+        const IV = ν*cosφ;
+        const V = (ν/6)*cos3φ*(ν/ρ-tan2φ);
+        const VI = (ν/120) * cos5φ * (5 - 18*tan2φ + tan4φ + 14*η2 - 58*tan2φ*η2);
+
+        const Δλ = λ-λ0;
+        const Δλ2 = Δλ*Δλ, Δλ3 = Δλ2*Δλ, Δλ4 = Δλ3*Δλ, Δλ5 = Δλ4*Δλ, Δλ6 = Δλ5*Δλ;
+
+        let N = I + II*Δλ2 + III*Δλ4 + IIIA*Δλ6;
+        let E = E0 + IV*Δλ + V*Δλ3 + VI*Δλ5;
+
+        N = Number(N.toFixed(3)); // round to mm precision
+        E = Number(E.toFixed(3));
+
+        try {
+            return new IeGridRef(E, N); // note: gets truncated to SW corner of 1m grid square
+        } catch (e) {
+            throw new Error(`${e.message} from (${point.lat.toFixed(6)},${point.lon.toFixed(6)}).toOsGrid()`);
+        }
+    }
+
+
+    /**
+     * Override LatLonEllipsoidal.convertDatum() with version which returns LatLon_IeGridRef.
+     */
+    convertDatum(toDatum) {
+        const osieED = super.convertDatum(toDatum); // returns LatLonEllipsoidal_Datum
+        const osieOSGR = new LatLon_IeGridRef(osieED.lat, osieED.lon, osieED.height, osieED.datum);
+        return osieOSGR;
+    }
+
+}
+
+
+/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  */
+
+export { IeGridRef as default, LatLon_IeGridRef as LatLon, Dms };

static/vendor/js/modules/geodesy/osgridref.js

@@ -0,0 +1,348 @@
+/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  */
+/* Ordnance Survey Grid Reference functions                           (c) Chris Veness 2005-2021  */
+/*                                                                                   MIT Licence  */
+/* www.movable-type.co.uk/scripts/latlong-gridref.html                                            */
+/* www.movable-type.co.uk/scripts/geodesy-library.html#osgridref                                  */
+/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  */
+
+import LatLonEllipsoidal, { Dms } from 'https://cdn.jsdelivr.net/npm/geodesy@2/latlon-ellipsoidal-datum.js';
+
+
+/**
+ * Ordnance Survey OSGB grid references provide geocoordinate references for UK mapping purposes.
+ *
+ * Formulation implemented here due to Thomas, Redfearn, etc is as published by OS, but is inferior
+ * to Krüger as used by e.g. Karney 2011.
+ *
+ * www.ordnancesurvey.co.uk/documents/resources/guide-coordinate-systems-great-britain.pdf.
+ *
+ * Note OSGB grid references cover Great Britain only; Ireland and the Channel Islands have their
+ * own references.
+ *
+ * Note that these formulae are based on ellipsoidal calculations, and according to the OS are
+ * accurate to about 4–5 metres – for greater accuracy, a geoid-based transformation (OSTN15) must
+ * be used.
+ */
+
+/*
+ * Converted 2015 to work with WGS84 by default, OSGB36 as option;
+ * www.ordnancesurvey.co.uk/blog/2014/12/confirmation-on-changes-to-latitude-and-longitude
+ */
+
+
+/* OsGridRef  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
+
+
+const nationalGrid = {
+    trueOrigin:  { lat: 49, lon: -2 },                 // true origin of grid 49°N,2°W on OSGB36 datum
+    falseOrigin: { easting: -400e3, northing: 100e3 }, // easting & northing of false origin, metres from true origin
+    scaleFactor: 0.9996012717,                         // scale factor on central meridian
+    ellipsoid:   LatLonEllipsoidal.ellipsoids.Airy1830,
+};
+// note Irish National Grid uses t/o 53°30′N, 8°W, f/o 200kmW, 250kmS, scale factor 1.000035, on Airy 1830 Modified ellipsoid
+
+
+/**
+ * OS Grid References with methods to parse and convert them to latitude/longitude points.
+ */
+class OsGridRef {
+
+    /**
+     * Creates an OsGridRef object.
+     *
+     * @param {number} easting - Easting in metres from OS Grid false origin.
+     * @param {number} northing - Northing in metres from OS Grid false origin.
+     *
+     * @example
+     *   import OsGridRef from '/js/geodesy/osgridref.js';
+     *   const gridref = new OsGridRef(651409, 313177);
+     */
+    constructor(easting, northing) {
+        this.easting = Number(easting);
+        this.northing = Number(northing);
+
+        if (isNaN(easting)  || this.easting<0  || this.easting>700e3) throw new RangeError(`invalid easting ‘${easting}’`);
+        if (isNaN(northing) || this.northing<0 || this.northing>1300e3) throw new RangeError(`invalid northing ‘${northing}’`);
+    }
+
+
+    /**
+     * Converts ‘this’ Ordnance Survey Grid Reference easting/northing coordinate to latitude/longitude
+     * (SW corner of grid square).
+     *
+     * While OS Grid References are based on OSGB-36, the Ordnance Survey have deprecated the use of
+     * OSGB-36 for latitude/longitude coordinates (in favour of WGS-84), hence this function returns
+     * WGS-84 by default, with OSGB-36 as an option. See www.ordnancesurvey.co.uk/blog/2014/12/2.
+     *
+     * Note formulation implemented here due to Thomas, Redfearn, etc is as published by OS, but is
+     * inferior to Krüger as used by e.g. Karney 2011.
+     *
+     * @param   {LatLon.datum} [datum=WGS84] - Datum to convert grid reference into.
+     * @returns {LatLon}       Latitude/longitude of supplied grid reference.
+     *
+     * @example
+     *   const gridref = new OsGridRef(651409.903, 313177.270);
+     *   const pWgs84 = gridref.toLatLon();                    // 52°39′28.723″N, 001°42′57.787″E
+     *   // to obtain (historical) OSGB36 lat/lon point:
+     *   const pOsgb = gridref.toLatLon(LatLon.datums.OSGB36); // 52°39′27.253″N, 001°43′04.518″E
+     */
+    toLatLon(datum=LatLonEllipsoidal.datums.WGS84) {
+        const { easting: E, northing: N } = this;
+
+        const { a, b } = nationalGrid.ellipsoid;            // a = 6377563.396, b = 6356256.909
+        const φ0 = nationalGrid.trueOrigin.lat.toRadians(); // latitude of true origin, 49°N
+        const λ0 = nationalGrid.trueOrigin.lon.toRadians(); // longitude of true origin, 2°W
+        const E0 = -nationalGrid.falseOrigin.easting;       // easting of true origin, 400km
+        const N0 = -nationalGrid.falseOrigin.northing;      // northing of true origin, -100km
+        const F0 = nationalGrid.scaleFactor;                // 0.9996012717
+
+        const e2 = 1 - (b*b)/(a*a);                         // eccentricity squared
+        const n = (a-b)/(a+b), n2 = n*n, n3 = n*n*n;        // n, n², n³
+
+        let φ=φ0, M=0;
+        do {
+            φ = (N-N0-M)/(a*F0) + φ;
+
+            const Ma = (1 + n + (5/4)*n2 + (5/4)*n3) * (φ-φ0);
+            const Mb = (3*n + 3*n2 + (21/8)*n3) * Math.sin(φ-φ0) * Math.cos(φ+φ0);
+            const Mc = ((15/8)*n2 + (15/8)*n3) * Math.sin(2*(φ-φ0)) * Math.cos(2*(φ+φ0));
+            const Md = (35/24)*n3 * Math.sin(3*(φ-φ0)) * Math.cos(3*(φ+φ0));
+            M = b * F0 * (Ma - Mb + Mc - Md);                // meridional arc
+
+        } while (Math.abs(N-N0-M) >= 0.00001);  // ie until < 0.01mm
+
+        const cosφ = Math.cos(φ), sinφ = Math.sin(φ);
+        const ν = a*F0/Math.sqrt(1-e2*sinφ*sinφ);            // nu = transverse radius of curvature
+        const ρ = a*F0*(1-e2)/Math.pow(1-e2*sinφ*sinφ, 1.5); // rho = meridional radius of curvature
+        const η2 = ν/ρ-1;                                    // eta = ?
+
+        const tanφ = Math.tan(φ);
+        const tan2φ = tanφ*tanφ, tan4φ = tan2φ*tan2φ, tan6φ = tan4φ*tan2φ;
+        const secφ = 1/cosφ;
+        const ν3 = ν*ν*ν, ν5 = ν3*ν*ν, ν7 = ν5*ν*ν;
+        const VII = tanφ/(2*ρ*ν);
+        const VIII = tanφ/(24*ρ*ν3)*(5+3*tan2φ+η2-9*tan2φ*η2);
+        const IX = tanφ/(720*ρ*ν5)*(61+90*tan2φ+45*tan4φ);
+        const X = secφ/ν;
+        const XI = secφ/(6*ν3)*(ν/ρ+2*tan2φ);
+        const XII = secφ/(120*ν5)*(5+28*tan2φ+24*tan4φ);
+        const XIIA = secφ/(5040*ν7)*(61+662*tan2φ+1320*tan4φ+720*tan6φ);
+
+        const dE = (E-E0), dE2 = dE*dE, dE3 = dE2*dE, dE4 = dE2*dE2, dE5 = dE3*dE2, dE6 = dE4*dE2, dE7 = dE5*dE2;
+        φ = φ - VII*dE2 + VIII*dE4 - IX*dE6;
+        const λ = λ0 + X*dE - XI*dE3 + XII*dE5 - XIIA*dE7;
+
+        let point = new LatLon_OsGridRef(φ.toDegrees(), λ.toDegrees(), 0, LatLonEllipsoidal.datums.OSGB36);
+
+        if (datum != LatLonEllipsoidal.datums.OSGB36) {
+            // if point is required in datum other than OSGB36, convert it
+            point = point.convertDatum(datum);
+            // convertDatum() gives us a LatLon: convert to LatLon_OsGridRef which includes toOsGrid()
+            point = new LatLon_OsGridRef(point.lat, point.lon, point.height, point.datum);
+        }
+
+        return point;
+    }
+
+
+    /**
+     * Parses grid reference to OsGridRef object.
+     *
+     * Accepts standard grid references (eg 'SU 387 148'), with or without whitespace separators, from
+     * two-digit references up to 10-digit references (1m × 1m square), or fully numeric comma-separated
+     * references in metres (eg '438700,114800').
+     *
+     * @param   {string}    gridref - Standard format OS Grid Reference.
+     * @returns {OsGridRef} Numeric version of grid reference in metres from false origin (SW corner of
+     *   supplied grid square).
+     * @throws  {Error}     Invalid grid reference.
+     *
+     * @example
+     *   const grid = OsGridRef.parse('TG 51409 13177'); // grid: { easting: 651409, northing: 313177 }
+     */
+    static parse(gridref) {
+        gridref = String(gridref).trim();
+
+        // check for fully numeric comma-separated gridref format
+        let match = gridref.match(/^(\d+),\s*(\d+)$/);
+        if (match) return new OsGridRef(match[1], match[2]);
+
+        // validate format
+        match = gridref.match(/^[HNOST][ABCDEFGHJKLMNOPQRSTUVWXYZ]\s*[0-9]+\s*[0-9]+$/i);
+        if (!match) throw new Error(`invalid grid reference ‘${gridref}’`);
+
+        // get numeric values of letter references, mapping A->0, B->1, C->2, etc:
+        let l1 = gridref.toUpperCase().charCodeAt(0) - 'A'.charCodeAt(0); // 500km square
+        let l2 = gridref.toUpperCase().charCodeAt(1) - 'A'.charCodeAt(0); // 100km square
+        // shuffle down letters after 'I' since 'I' is not used in grid:
+        if (l1 > 7) l1--;
+        if (l2 > 7) l2--;
+
+        // convert grid letters into 100km-square indexes from false origin (grid square SV):
+        const e100km = ((l1 - 2) % 5) * 5 + (l2 % 5);
+        const n100km = (19 - Math.floor(l1 / 5) * 5) - Math.floor(l2 / 5);
+
+        // skip grid letters to get numeric (easting/northing) part of ref
+        let en = gridref.slice(2).trim().split(/\s+/);
+        // if e/n not whitespace separated, split half way
+        if (en.length == 1) en = [ en[0].slice(0, en[0].length / 2), en[0].slice(en[0].length / 2) ];
+
+        // validation
+        if (en[0].length != en[1].length) throw new Error(`invalid grid reference ‘${gridref}’`);
+
+        // standardise to 10-digit refs (metres)
+        en[0] = en[0].padEnd(5, '0');
+        en[1] = en[1].padEnd(5, '0');
+
+        const e = e100km + en[0];
+        const n = n100km + en[1];
+
+        return new OsGridRef(e, n);
+    }
+
+
+    /**
+     * Converts ‘this’ numeric grid reference to standard OS Grid Reference.
+     *
+     * @param   {number} [digits=10] - Precision of returned grid reference (10 digits = metres);
+     *   digits=0 will return grid reference in numeric format.
+     * @returns {string} This grid reference in standard format.
+     *
+     * @example
+     *   const gridref = new OsGridRef(651409, 313177).toString(8); // 'TG 5140 1317'
+     *   const gridref = new OsGridRef(651409, 313177).toString(0); // '651409,313177'
+     */
+    toString(digits=10) {
+        if (![ 0,2,4,6,8,10,12,14,16 ].includes(Number(digits))) throw new RangeError(`invalid precision ‘${digits}’`); // eslint-disable-line comma-spacing
+
+        let { easting: e, northing: n } = this;
+
+        // use digits = 0 to return numeric format (in metres) - note northing may be >= 1e7
+        if (digits == 0) {
+            const format = { useGrouping: false,  minimumIntegerDigits: 6, maximumFractionDigits: 3 };
+            const ePad = e.toLocaleString('en', format);
+            const nPad = n.toLocaleString('en', format);
+            return `${ePad},${nPad}`;
+        }
+
+        // get the 100km-grid indices
+        const e100km = Math.floor(e / 100000), n100km = Math.floor(n / 100000);
+
+        // translate those into numeric equivalents of the grid letters
+        let l1 = (19 - n100km) - (19 - n100km) % 5 + Math.floor((e100km + 10) / 5);
+        let l2 = (19 - n100km) * 5 % 25 + e100km % 5;
+
+        // compensate for skipped 'I' and build grid letter-pairs
+        if (l1 > 7) l1++;
+        if (l2 > 7) l2++;
+        const letterPair = String.fromCharCode(l1 + 'A'.charCodeAt(0), l2 + 'A'.charCodeAt(0));
+
+        // strip 100km-grid indices from easting & northing, and reduce precision
+        e = Math.floor((e % 100000) / Math.pow(10, 5 - digits / 2));
+        n = Math.floor((n % 100000) / Math.pow(10, 5 - digits / 2));
+
+        // pad eastings & northings with leading zeros
+        e = e.toString().padStart(digits/2, '0');
+        n = n.toString().padStart(digits/2, '0');
+
+        return `${letterPair} ${e} ${n}`;
+    }
+
+}
+
+
+/* LatLon_OsGridRef - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
+
+
+/**
+ * Extends LatLon class with method to convert LatLon point to OS Grid Reference.
+ *
+ * @extends LatLonEllipsoidal
+ */
+class LatLon_OsGridRef extends LatLonEllipsoidal {
+
+    /**
+     * Converts latitude/longitude to Ordnance Survey grid reference easting/northing coordinate.
+     *
+     * @returns {OsGridRef} OS Grid Reference easting/northing.
+     *
+     * @example
+     *   const grid = new LatLon(52.65798, 1.71605).toOsGrid(); // TG 51409 13177
+     *   // for conversion of (historical) OSGB36 latitude/longitude point:
+     *   const grid = new LatLon(52.65798, 1.71605).toOsGrid(LatLon.datums.OSGB36);
+     */
+    toOsGrid() {
+        // if necessary convert to OSGB36 first
+        const point = this.datum == LatLonEllipsoidal.datums.OSGB36
+            ? this
+            : this.convertDatum(LatLonEllipsoidal.datums.OSGB36);
+
+        const φ = point.lat.toRadians();
+        const λ = point.lon.toRadians();
+
+        const { a, b } = nationalGrid.ellipsoid;            // a = 6377563.396, b = 6356256.909
+        const φ0 = nationalGrid.trueOrigin.lat.toRadians(); // latitude of true origin, 49°N
+        const λ0 = nationalGrid.trueOrigin.lon.toRadians(); // longitude of true origin, 2°W
+        const E0 = -nationalGrid.falseOrigin.easting;       // easting of true origin, 400km
+        const N0 = -nationalGrid.falseOrigin.northing;      // northing of true origin, -100km
+        const F0 = nationalGrid.scaleFactor;                // 0.9996012717
+
+        const e2 = 1 - (b*b)/(a*a);                          // eccentricity squared
+        const n = (a-b)/(a+b), n2 = n*n, n3 = n*n*n;         // n, n², n³
+
+        const cosφ = Math.cos(φ), sinφ = Math.sin(φ);
+        const ν = a*F0/Math.sqrt(1-e2*sinφ*sinφ);            // nu = transverse radius of curvature
+        const ρ = a*F0*(1-e2)/Math.pow(1-e2*sinφ*sinφ, 1.5); // rho = meridional radius of curvature
+        const η2 = ν/ρ-1;                                    // eta = ?
+
+        const Ma = (1 + n + (5/4)*n2 + (5/4)*n3) * (φ-φ0);
+        const Mb = (3*n + 3*n2 + (21/8)*n3) * Math.sin(φ-φ0) * Math.cos(φ+φ0);
+        const Mc = ((15/8)*n2 + (15/8)*n3) * Math.sin(2*(φ-φ0)) * Math.cos(2*(φ+φ0));
+        const Md = (35/24)*n3 * Math.sin(3*(φ-φ0)) * Math.cos(3*(φ+φ0));
+        const M = b * F0 * (Ma - Mb + Mc - Md);              // meridional arc
+
+        const cos3φ = cosφ*cosφ*cosφ;
+        const cos5φ = cos3φ*cosφ*cosφ;
+        const tan2φ = Math.tan(φ)*Math.tan(φ);
+        const tan4φ = tan2φ*tan2φ;
+
+        const I = M + N0;
+        const II = (ν/2)*sinφ*cosφ;
+        const III = (ν/24)*sinφ*cos3φ*(5-tan2φ+9*η2);
+        const IIIA = (ν/720)*sinφ*cos5φ*(61-58*tan2φ+tan4φ);
+        const IV = ν*cosφ;
+        const V = (ν/6)*cos3φ*(ν/ρ-tan2φ);
+        const VI = (ν/120) * cos5φ * (5 - 18*tan2φ + tan4φ + 14*η2 - 58*tan2φ*η2);
+
+        const Δλ = λ-λ0;
+        const Δλ2 = Δλ*Δλ, Δλ3 = Δλ2*Δλ, Δλ4 = Δλ3*Δλ, Δλ5 = Δλ4*Δλ, Δλ6 = Δλ5*Δλ;
+
+        let N = I + II*Δλ2 + III*Δλ4 + IIIA*Δλ6;
+        let E = E0 + IV*Δλ + V*Δλ3 + VI*Δλ5;
+
+        N = Number(N.toFixed(3)); // round to mm precision
+        E = Number(E.toFixed(3));
+
+        try {
+            return new OsGridRef(E, N); // note: gets truncated to SW corner of 1m grid square
+        } catch (e) {
+            throw new Error(`${e.message} from (${point.lat.toFixed(6)},${point.lon.toFixed(6)}).toOsGrid()`);
+        }
+    }
+
+
+    /**
+     * Override LatLonEllipsoidal.convertDatum() with version which returns LatLon_OsGridRef.
+     */
+    convertDatum(toDatum) {
+        const osgbED = super.convertDatum(toDatum); // returns LatLonEllipsoidal_Datum
+        const osgbOSGR = new LatLon_OsGridRef(osgbED.lat, osgbED.lon, osgbED.height, osgbED.datum);
+        return osgbOSGR;
+    }
+
+}
+
+
+/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  */
+
+export { OsGridRef as default, LatLon_OsGridRef as LatLon, Dms };

static/vendor/js/modules/geodesy/utm_ci.js

@@ -0,0 +1,413 @@
+/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  */
+/* UTM / WGS-84 Conversion Functions                (c) Chris Veness 2014-2022 & Ian Renton 2025  */
+/*                                                                                   MIT Licence  */
+/* www.movable-type.co.uk/scripts/latlong-utm-mgrs.html                                           */
+/* www.movable-type.co.uk/scripts/geodesy-library.html#utm                                        */
+/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  */
+
+/* eslint-disable indent */
+
+import LatLonEllipsoidal, { Dms } from 'https://cdn.jsdelivr.net/npm/geodesy@2/latlon-ellipsoidal-datum.js';
+
+
+/**
+ * The Universal Transverse Mercator (UTM) system is a 2-dimensional Cartesian coordinate system
+ * providing locations on the surface of the Earth.
+ *
+ * UTM is a set of 60 transverse Mercator projections, normally based on the WGS-84 ellipsoid.
+ * Within each zone, coordinates are represented as eastings and northings, measures in metres; e.g.
+ * ‘31 N 448251 5411932’.
+ *
+ * This method based on Karney 2011 ‘Transverse Mercator with an accuracy of a few nanometers’,
+ * building on Krüger 1912 ‘Konforme Abbildung des Erdellipsoids in der Ebene’.
+ *
+ * @module utm
+ */
+
+
+/* Utm  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
+
+
+/**
+ * UTM coordinates, with functions to parse them and convert them to LatLon points.
+ */
+class Utm {
+
+    /**
+     * Creates a Utm coordinate object comprising zone, hemisphere, easting, northing on a given
+     * datum (normally WGS84).
+     *
+     * @param  {number}        zone - UTM 6° longitudinal zone (1..60 covering 180°W..180°E).
+     * @param  {string}        hemisphere - N for northern hemisphere, S for southern hemisphere.
+     * @param  {number}        easting - Easting in metres from false easting (-500km from central meridian).
+     * @param  {number}        northing - Northing in metres from equator (N) or from false northing -10,000km (S).
+     * @param  {LatLon.datums} [datum=WGS84] - Datum UTM coordinate is based on.
+     * @param  {number}        [convergence=null] - Meridian convergence (bearing of grid north
+     *                         clockwise from true north), in degrees.
+     * @param  {number}        [scale=null] - Grid scale factor.
+     * @params {boolean=true}  verifyEN - Check easting/northing is within 'normal' values (may be
+     *                         suppressed for extended coherent coordinates or alternative datums
+     *                         e.g. ED50 (epsg.io/23029).
+     * @throws {TypeError} Invalid UTM coordinate.
+     *
+     * @example
+     *   import Utm from '/js/geodesy/utm.js';
+     *   const utmCoord = new Utm(31, 'N', 448251, 5411932);
+     */
+    constructor(zone, hemisphere, easting, northing, datum=LatLonEllipsoidal.datums.WGS84, convergence=null, scale=null, verifyEN=true) {
+        if (!(1<=zone && zone<=60)) throw new RangeError(`invalid UTM zone ‘${zone}’`);
+        if (zone != parseInt(zone)) throw new RangeError(`invalid UTM zone ‘${zone}’`);
+        if (typeof hemisphere != 'string' || !hemisphere.match(/[NS]/i)) throw new RangeError(`invalid UTM hemisphere ‘${hemisphere}’`);
+        if (verifyEN) { // (rough) range-check of E/N values
+            if (!(0<=easting && easting<=1000e3)) throw new RangeError(`invalid UTM easting ‘${easting}’`);
+            if (hemisphere.toUpperCase()=='N' && !(0<=northing && northing<9329006)) throw new RangeError(`invalid UTM northing ‘${northing}’`);
+            if (hemisphere.toUpperCase()=='S' && !(1116914<northing && northing<=10000e3)) throw new RangeError(`invalid UTM northing ‘${northing}’`);
+        }
+        if (!datum || datum.ellipsoid==undefined) throw new TypeError(`unrecognised datum ‘${datum}’`);
+
+        this.zone = Number(zone);
+        this.hemisphere = hemisphere.toUpperCase();
+        this.easting = Number(easting);
+        this.northing = Number(northing);
+        this.datum = datum;
+        this.convergence = convergence===null ? null : Number(convergence);
+        this.scale = scale===null ? null : Number(scale);
+    }
+
+
+    /**
+     * Converts UTM zone/easting/northing coordinate to latitude/longitude.
+     *
+     * Implements Karney’s method, using Krüger series to order n⁶, giving results accurate to 5nm
+     * for distances up to 3900km from the central meridian.
+     *
+     * @param   {Utm} utmCoord - UTM coordinate to be converted to latitude/longitude.
+     * @returns {LatLon} Latitude/longitude of supplied grid reference.
+     *
+     * @example
+     *   const grid = new Utm(31, 'N', 448251.795, 5411932.678);
+     *   const latlong = grid.toLatLon(); // 48°51′29.52″N, 002°17′40.20″E
+     */
+    toLatLon() {
+        const { zone: z, hemisphere: h } = this;
+
+        const falseEasting = 500e3, falseNorthing = 10000e3;
+
+        const { a, f } = this.datum.ellipsoid; // WGS-84: a = 6378137, f = 1/298.257223563;
+
+        const k0 = 0.9996; // UTM scale on the central meridian
+
+        const x = this.easting - falseEasting;                            // make x ± relative to central meridian
+        const y = h=='S' ? this.northing - falseNorthing : this.northing; // make y ± relative to equator
+
+        // ---- from Karney 2011 Eq 15-22, 36:
+
+        const e = Math.sqrt(f*(2-f)); // eccentricity
+        const n = f / (2 - f);        // 3rd flattening
+        const n2 = n*n, n3 = n*n2, n4 = n*n3, n5 = n*n4, n6 = n*n5;
+
+        const A = a/(1+n) * (1 + 1/4*n2 + 1/64*n4 + 1/256*n6); // 2πA is the circumference of a meridian
+
+        const η = x / (k0*A);
+        const ξ = y / (k0*A);
+
+        const β = [ null, // note β is one-based array (6th order Krüger expressions)
+            1/2*n - 2/3*n2 + 37/96*n3 -    1/360*n4 -   81/512*n5 +    96199/604800*n6,
+                   1/48*n2 +  1/15*n3 - 437/1440*n4 +   46/105*n5 - 1118711/3870720*n6,
+                            17/480*n3 -   37/840*n4 - 209/4480*n5 +      5569/90720*n6,
+                                     4397/161280*n4 -   11/504*n5 -  830251/7257600*n6,
+                                                   4583/161280*n5 -  108847/3991680*n6,
+                                                                 20648693/638668800*n6 ];
+
+        let ξʹ = ξ;
+        for (let j=1; j<=6; j++) ξʹ -= β[j] * Math.sin(2*j*ξ) * Math.cosh(2*j*η);
+
+        let ηʹ = η;
+        for (let j=1; j<=6; j++) ηʹ -= β[j] * Math.cos(2*j*ξ) * Math.sinh(2*j*η);
+
+        const sinhηʹ = Math.sinh(ηʹ);
+        const sinξʹ = Math.sin(ξʹ), cosξʹ = Math.cos(ξʹ);
+
+        const τʹ = sinξʹ / Math.sqrt(sinhηʹ*sinhηʹ + cosξʹ*cosξʹ);
+
+        let δτi = null;
+        let τi = τʹ;
+        do {
+            const σi = Math.sinh(e*Math.atanh(e*τi/Math.sqrt(1+τi*τi)));
+            const τiʹ = τi * Math.sqrt(1+σi*σi) - σi * Math.sqrt(1+τi*τi);
+            δτi = (τʹ - τiʹ)/Math.sqrt(1+τiʹ*τiʹ)
+                * (1 + (1-e*e)*τi*τi) / ((1-e*e)*Math.sqrt(1+τi*τi));
+            τi += δτi;
+        } while (Math.abs(δτi) > 1e-12); // using IEEE 754 δτi -> 0 after 2-3 iterations
+        // note relatively large convergence test as δτi toggles on ±1.12e-16 for eg 31 N 400000 5000000
+        const τ = τi;
+
+        const φ = Math.atan(τ);
+
+        let λ = Math.atan2(sinhηʹ, cosξʹ);
+
+        // ---- convergence: Karney 2011 Eq 26, 27
+
+        let p = 1;
+        for (let j=1; j<=6; j++) p -= 2*j*β[j] * Math.cos(2*j*ξ) * Math.cosh(2*j*η);
+        let q = 0;
+        for (let j=1; j<=6; j++) q += 2*j*β[j] * Math.sin(2*j*ξ) * Math.sinh(2*j*η);
+
+        const γʹ = Math.atan(Math.tan(ξʹ) * Math.tanh(ηʹ));
+        const γʺ = Math.atan2(q, p);
+
+        const γ = γʹ + γʺ;
+
+        // ---- scale: Karney 2011 Eq 28
+
+        const sinφ = Math.sin(φ);
+        const kʹ = Math.sqrt(1 - e*e*sinφ*sinφ) * Math.sqrt(1 + τ*τ) * Math.sqrt(sinhηʹ*sinhηʹ + cosξʹ*cosξʹ);
+        const kʺ = A / a / Math.sqrt(p*p + q*q);
+
+        const k = k0 * kʹ * kʺ;
+
+        // ------------
+
+        const λ0 = ((z-1)*6 - 180 + 3).toRadians(); // longitude of central meridian
+        λ += λ0; // move λ from zonal to global coordinates
+
+        // round to reasonable precision
+        const lat = Number(φ.toDegrees().toFixed(14)); // nm precision (1nm = 10^-14°)
+        const lon = Number(λ.toDegrees().toFixed(14)); // (strictly lat rounding should be φ⋅cosφ!)
+        const convergence = Number(γ.toDegrees().toFixed(9));
+        const scale = Number(k.toFixed(12));
+
+        const latLong = new LatLon_Utm(lat, lon, 0, this.datum);
+        // ... and add the convergence and scale into the LatLon object ... wonderful JavaScript!
+        latLong.convergence = convergence;
+        latLong.scale = scale;
+
+        return latLong;
+    }
+
+    /**
+     * Parses a Channel Islands (WA/WV) grid reference.
+     */
+    static parseChannelIslandGrid(gridref) {
+        // validate format
+        let match = gridref.match(/^W[AV]\s*[0-9]+\s*[0-9]+$/i);
+        if (!match) throw new Error(`invalid grid reference ‘${gridref}’`);
+
+        // skip grid letters to get numeric (easting/northing) part of ref
+        let en = gridref.slice(2).trim().split(/\s+/);
+        // if e/n not whitespace separated, split half way
+        if (en.length == 1) en = [ en[0].slice(0, en[0].length / 2), en[0].slice(en[0].length / 2) ];
+
+        // validation
+        if (en[0].length != en[1].length) throw new Error(`invalid grid reference ‘${gridref}’`);
+
+        // standardise to 10-digit refs (metres)
+        en[0] = en[0].padEnd(5, '0');
+        en[1] = en[1].padEnd(5, '0');
+
+        let utmCoord = "30 N ";
+        const e = 5 + en[0];
+        utmCoord += e + " ";
+        if (gridref.substring(0, 2) === "WA") {
+            const n = 55 + en[1];
+            utmCoord += n;
+        } else if (gridref.substring(0, 2) === "WV") {
+            const n = 54 + en[1];
+            utmCoord += n;
+        }
+
+        return Utm.parse(utmCoord);
+    }
+
+
+    /**
+     * Parses string representation of UTM coordinate.
+     *
+     * A UTM coordinate comprises (space-separated)
+     *  - zone
+     *  - hemisphere
+     *  - easting
+     *  - northing.
+     *
+     * @param   {string} utmCoord - UTM coordinate (WGS 84).
+     * @param   {Datum}  [datum=WGS84] - Datum coordinate is defined in (default WGS 84).
+     * @returns {Utm} Parsed UTM coordinate.
+     * @throws  {TypeError} Invalid UTM coordinate.
+     *
+     * @example
+     *   const utmCoord = Utm.parse('31 N 448251 5411932');
+     *   // utmCoord: {zone: 31, hemisphere: 'N', easting: 448251, northing: 5411932 }
+     */
+    static parse(utmCoord, datum=LatLonEllipsoidal.datums.WGS84) {
+        // match separate elements (separated by whitespace)
+        utmCoord = utmCoord.trim().match(/\S+/g);
+
+        if (utmCoord==null || utmCoord.length!=4) throw new Error(`invalid UTM coordinate ‘${utmCoord}’`);
+
+        const zone = utmCoord[0], hemisphere = utmCoord[1], easting = utmCoord[2], northing = utmCoord[3];
+
+        return new this(zone, hemisphere, easting, northing, datum); // 'new this' as may return subclassed types
+    }
+
+
+    /**
+     * Returns a string representation of a UTM coordinate.
+     *
+     * To distinguish from MGRS grid zone designators, a space is left between the zone and the
+     * hemisphere.
+     *
+     * Note that UTM coordinates get rounded, not truncated (unlike MGRS grid references).
+     *
+     * @param   {number} [digits=0] - Number of digits to appear after the decimal point (3 ≡ mm).
+     * @returns {string} A string representation of the coordinate.
+     *
+     * @example
+     *   const utm = new Utm('31', 'N', 448251, 5411932).toString(4);  // 31 N 448251.0000 5411932.0000
+     */
+    toString(digits=0) {
+
+        const z = this.zone.toString().padStart(2, '0');
+        const h = this.hemisphere;
+        const e = this.easting.toFixed(digits);
+        const n = this.northing.toFixed(digits);
+
+        return `${z} ${h} ${e} ${n}`;
+    }
+
+}
+
+
+/* LatLon_Utm - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
+
+
+/**
+ * Extends LatLon with method to convert LatLon points to UTM coordinates.
+ *
+ * @extends LatLon
+ */
+class LatLon_Utm extends LatLonEllipsoidal {
+
+    /**
+     * Converts latitude/longitude to UTM coordinate.
+     *
+     * Implements Karney’s method, using Krüger series to order n⁶, giving results accurate to 5nm
+     * for distances up to 3900km from the central meridian.
+     *
+     * @param   {number} [zoneOverride] - Use specified zone rather than zone within which point lies;
+     *          note overriding the UTM zone has the potential to result in negative eastings, and
+     *          perverse results within Norway/Svalbard exceptions.
+     * @returns {Utm} UTM coordinate.
+     * @throws  {TypeError} Latitude outside UTM limits.
+     *
+     * @example
+     *   const latlong = new LatLon(48.8582, 2.2945);
+     *   const utmCoord = latlong.toUtm(); // 31 N 448252 5411933
+     */
+    toUtm(zoneOverride=undefined) {
+        if (!(-80<=this.lat && this.lat<=84)) throw new RangeError(`latitude ‘${this.lat}’ outside UTM limits`);
+
+        const falseEasting = 500e3, falseNorthing = 10000e3;
+
+        let zone = zoneOverride || Math.floor((this.lon+180)/6) + 1; // longitudinal zone
+        let λ0 = ((zone-1)*6 - 180 + 3).toRadians(); // longitude of central meridian
+
+        // ---- handle Norway/Svalbard exceptions
+        // grid zones are 8° tall; 0°N is offset 10 into latitude bands array
+        const mgrsLatBands = 'CDEFGHJKLMNPQRSTUVWXX'; // X is repeated for 80-84°N
+        const latBand = mgrsLatBands.charAt(Math.floor(this.lat/8+10));
+        // adjust zone & central meridian for Norway
+        if (zone==31 && latBand=='V' && this.lon>= 3) { zone++; λ0 += (6).toRadians(); }
+        // adjust zone & central meridian for Svalbard
+        if (zone==32 && latBand=='X' && this.lon<  9) { zone--; λ0 -= (6).toRadians(); }
+        if (zone==32 && latBand=='X' && this.lon>= 9) { zone++; λ0 += (6).toRadians(); }
+        if (zone==34 && latBand=='X' && this.lon< 21) { zone--; λ0 -= (6).toRadians(); }
+        if (zone==34 && latBand=='X' && this.lon>=21) { zone++; λ0 += (6).toRadians(); }
+        if (zone==36 && latBand=='X' && this.lon< 33) { zone--; λ0 -= (6).toRadians(); }
+        if (zone==36 && latBand=='X' && this.lon>=33) { zone++; λ0 += (6).toRadians(); }
+
+        const φ = this.lat.toRadians();      // latitude ± from equator
+        const λ = this.lon.toRadians() - λ0; // longitude ± from central meridian
+
+        // allow alternative ellipsoid to be specified
+        const ellipsoid = this.datum ? this.datum.ellipsoid : LatLonEllipsoidal.ellipsoids.WGS84;
+        const { a, f } = ellipsoid; // WGS-84: a = 6378137, f = 1/298.257223563;
+
+        const k0 = 0.9996; // UTM scale on the central meridian
+
+        // ---- easting, northing: Karney 2011 Eq 7-14, 29, 35:
+
+        const e = Math.sqrt(f*(2-f)); // eccentricity
+        const n = f / (2 - f);        // 3rd flattening
+        const n2 = n*n, n3 = n*n2, n4 = n*n3, n5 = n*n4, n6 = n*n5;
+
+        const cosλ = Math.cos(λ), sinλ = Math.sin(λ), tanλ = Math.tan(λ);
+
+        const τ = Math.tan(φ); // τ ≡ tanφ, τʹ ≡ tanφʹ; prime (ʹ) indicates angles on the conformal sphere
+        const σ = Math.sinh(e*Math.atanh(e*τ/Math.sqrt(1+τ*τ)));
+
+        const τʹ = τ*Math.sqrt(1+σ*σ) - σ*Math.sqrt(1+τ*τ);
+
+        const ξʹ = Math.atan2(τʹ, cosλ);
+        const ηʹ = Math.asinh(sinλ / Math.sqrt(τʹ*τʹ + cosλ*cosλ));
+
+        const A = a/(1+n) * (1 + 1/4*n2 + 1/64*n4 + 1/256*n6); // 2πA is the circumference of a meridian
+
+        const α = [ null, // note α is one-based array (6th order Krüger expressions)
+            1/2*n - 2/3*n2 + 5/16*n3 +   41/180*n4 -     127/288*n5 +      7891/37800*n6,
+                  13/48*n2 -  3/5*n3 + 557/1440*n4 +     281/630*n5 - 1983433/1935360*n6,
+                           61/240*n3 -  103/140*n4 + 15061/26880*n5 +   167603/181440*n6,
+                                   49561/161280*n4 -     179/168*n5 + 6601661/7257600*n6,
+                                                     34729/80640*n5 - 3418889/1995840*n6,
+                                                                  212378941/319334400*n6 ];
+
+        let ξ = ξʹ;
+        for (let j=1; j<=6; j++) ξ += α[j] * Math.sin(2*j*ξʹ) * Math.cosh(2*j*ηʹ);
+
+        let η = ηʹ;
+        for (let j=1; j<=6; j++) η += α[j] * Math.cos(2*j*ξʹ) * Math.sinh(2*j*ηʹ);
+
+        let x = k0 * A * η;
+        let y = k0 * A * ξ;
+
+        // ---- convergence: Karney 2011 Eq 23, 24
+
+        let pʹ = 1;
+        for (let j=1; j<=6; j++) pʹ += 2*j*α[j] * Math.cos(2*j*ξʹ) * Math.cosh(2*j*ηʹ);
+        let qʹ = 0;
+        for (let j=1; j<=6; j++) qʹ += 2*j*α[j] * Math.sin(2*j*ξʹ) * Math.sinh(2*j*ηʹ);
+
+        const γʹ = Math.atan(τʹ / Math.sqrt(1+τʹ*τʹ)*tanλ);
+        const γʺ = Math.atan2(qʹ, pʹ);
+
+        const γ = γʹ + γʺ;
+
+        // ---- scale: Karney 2011 Eq 25
+
+        const sinφ = Math.sin(φ);
+        const kʹ = Math.sqrt(1 - e*e*sinφ*sinφ) * Math.sqrt(1 + τ*τ) / Math.sqrt(τʹ*τʹ + cosλ*cosλ);
+        const kʺ = A / a * Math.sqrt(pʹ*pʹ + qʹ*qʹ);
+
+        const k = k0 * kʹ * kʺ;
+
+        // ------------
+
+        // shift x/y to false origins
+        x = x + falseEasting;             // make x relative to false easting
+        if (y < 0) y = y + falseNorthing; // make y in southern hemisphere relative to false northing
+
+        // round to reasonable precision
+        x = Number(x.toFixed(9)); // nm precision
+        y = Number(y.toFixed(9)); // nm precision
+        const convergence = Number(γ.toDegrees().toFixed(9));
+        const scale = Number(k.toFixed(12));
+
+        const h = this.lat>=0 ? 'N' : 'S'; // hemisphere
+
+        return new Utm(zone, h, x, y, this.datum, convergence, scale, !!zoneOverride);
+    }
+}
+
+
+/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  */
+
+export { Utm as default, LatLon_Utm as LatLon, Dms };

templates/add_spot.html

@@ -76,7 +76,7 @@
 
 </div>
 
-<script src="/static/js/add-spot.js?v=1785434214"></script>
+<script src="/static/js/add-spot.js?v=1786175023"></script>
 <script>$(document).ready(function () {
     $("#nav-link-add-spot").addClass("active");
 }); <!-- highlight active page in nav --></script>

templates/alerts.html

@@ -75,7 +75,7 @@
 
 </div>
 
-<script src="/static/js/alerts.js?v=1785434213"></script>
+<script src="/static/js/alerts.js?v=1786175024"></script>
 <script>$(document).ready(function () {
     $("#nav-link-alerts").addClass("active");
 }); <!-- highlight active page in nav --></script>

templates/bands.html

@@ -75,8 +75,8 @@
 <script>
     let spotProvidersEnabledByDefault = {% raw json_encode(web_ui_options["spot-providers-enabled-by-default"]) %};
 </script>
-<script src="/static/js/spotsbandsandmap.js?v=1785434214"></script>
+<script src="/static/js/spotsbandsandmap.js?v=1786175023"></script>
-<script src="/static/js/bands.js?v=1785434214"></script>
+<script src="/static/js/bands.js?v=1786175023"></script>
 <script>$(document).ready(function () {
     $("#nav-link-bands").addClass("active");
 }); <!-- highlight active page in nav --></script>

templates/base.html

@@ -1,6 +1,6 @@
 {% extends "skeleton.html" %}
 {% block head_extra %}
-<link rel="stylesheet" href="/static/css/style.css?v=1785434213" type="text/css">
+<link rel="stylesheet" href="/static/css/style.css?v=1786175023" type="text/css">
 <link href="/static/vendor/css/bootstrap-5.3.8.min.css" rel="stylesheet">
 <link href="/static/vendor/css/fontawesome-6.7.2.min.css" rel="stylesheet">
 <link href="/static/vendor/css/solid-6.7.2.min.css" rel="stylesheet">
@@ -10,10 +10,10 @@
 <script src="/static/vendor/js/bootstrap-5.3.8.bundle.min.js"></script>
 <script src="/static/vendor/js/tinycolor2-1.6.0.min.js"></script>
 
-<script src="/static/js/utils.js?v=1785434213"></script>
+<script src="/static/js/utils.js?v=1786175023"></script>
-<script src="/static/js/ui-ham.js?v=1785434213"></script>
+<script src="/static/js/ui-ham.js?v=1786175023"></script>
-<script src="/static/js/geo.js?v=1785434213"></script>
+<script src="/static/js/geo.js?v=1786175023"></script>
-<script src="/static/js/common.js?v=1785434213"></script>
+<script src="/static/js/common.js?v=1786175023"></script>
 {% end %}
 {% block body %}
 <div class="container">

templates/conditions.html

@@ -284,7 +284,7 @@
 </div>
 
 <script src="/static/vendor/js/chart-4.4.9.umd.min.js"></script>
-<script src="/static/js/conditions.js?v=1785434213"></script>
+<script src="/static/js/conditions.js?v=1786175023"></script>
 <script>$(document).ready(function () {
     $("#nav-link-conditions").addClass("active");
 }); <!-- highlight active page in nav --></script>

templates/map.html

@@ -103,13 +103,13 @@
 <script src="/static/vendor/js/leaflet-maidenhead.js"></script>
 <script src="/static/vendor/js/leaflet-ituzones.js"></script>
 <script src="/static/vendor/js/leaflet-cqzones.js"></script>
-<script src="/static/vendor/js/leaflet-workedallbritainireland.js"></script>
+<script src="/static/vendor/js/leaflet-workedallbritainireland.js" type="module"></script>
 
 <script>
     let spotProvidersEnabledByDefault = {% raw json_encode(web_ui_options["spot-providers-enabled-by-default"]) %};
 </script>
-<script src="/static/js/spotsbandsandmap.js?v=1785434213"></script>
+<script src="/static/js/spotsbandsandmap.js?v=1786175024"></script>
-<script src="/static/js/map.js?v=1785434213"></script>
+<script src="/static/js/map.js?v=1786175024"></script>
 <script>$(document).ready(function () {
     $("#nav-link-map").addClass("active");
 }); <!-- highlight active page in nav --></script>

templates/spots.html

@@ -116,8 +116,8 @@
 <script>
     let spotProvidersEnabledByDefault = {% raw json_encode(web_ui_options["spot-providers-enabled-by-default"]) %};
 </script>
-<script src="/static/js/spotsbandsandmap.js?v=1785434213"></script>
+<script src="/static/js/spotsbandsandmap.js?v=1786175023"></script>
-<script src="/static/js/spots.js?v=1785434213"></script>
+<script src="/static/js/spots.js?v=1786175023"></script>
 <script>$(document).ready(function () {
     $("#nav-link-spots").addClass("active");
 }); <!-- highlight active page in nav --></script>

templates/status.html

@@ -59,7 +59,7 @@
     </div>
 </div>
 
-<script src="/static/js/status.js?v=1785434213"></script>
+<script src="/static/js/status.js?v=1786175023"></script>
 <script>
     $(document).ready(function () {
         $("#nav-link-status").addClass("active");