Various tweaks including getting GMA SIGs working properly

.gitignore

@@ -1,6 +1,7 @@
 /.venv
 __pycache__
 *.pyc
+/.data_store
 /.qrz_callsign_lookup_cache
 /sota_summit_data_cache.sqlite
 /gma_ref_info_cache.sqlite

README.md

@@ -2,15 +2,15 @@
 
 **Work in progress.**
 
-(S)pothole is a utility to aggregate "spots" from amateur radio DX clusters and xOTA spotting sites, and provide an open JSON API as well as a website to browse the data.
+Spothole is a utility to aggregate "spots" from amateur radio DX clusters and xOTA spotting sites, and provide an open JSON API as well as a website to browse the data.
 
 ![Screenshot](/images/screenshot.png)
 
-While there are several other web-based interfaces to DX clusters, and sites that aggregate spots from various outdoor activity programmes for amateur radio, (S)pothole differentiates itself by supporting a large number of data sources, and by being "API first" rather than just providing a web front-end. This allows other software to be built on top of it.
+While there are several other web-based interfaces to DX clusters, and sites that aggregate spots from various outdoor activity programmes for amateur radio, Spothole differentiates itself by supporting a large number of data sources, and by being "API first" rather than just providing a web front-end. This allows other software to be built on top of it.
 
 The API is deliberately well-defined with an OpenAPI specification and auto-generated API documentation. The API delivers spots in a consistent format regardless of the data source, freeing developers from needing to know how each individual data source presents its data.
 
-(S)pothole itself is also open source, Public Domain licenced code that anyone can take and modify.
+Spothole itself is also open source, Public Domain licenced code that anyone can take and modify.
 
 Supported data sources include DX Clusters, the Reverse Beacon Network (RBN), the APRS Internet Service (APRS-IS), POTA, SOTA, WWFF, GMA, WWBOTA, HEMA, and Parks 'n' Peaks.
 
@@ -125,7 +125,7 @@ Various approaches exist to writing your own client, but in general:
 
 ### Extending the server
 
-(S)pothole is designed to be easily extensible. If you want to write your own provider, simply add a module to the `providers` package containing your class. (Currently, in order to be loaded correctly, the module (file) name should be the same as the class name, but lower case.)
+Spothole is designed to be easily extensible. If you want to write your own provider, simply add a module to the `providers` package containing your class. (Currently, in order to be loaded correctly, the module (file) name should be the same as the class name, but lower case.)
 
 Your class should extend "Provider"; if it operates by polling an HTTP Server on a timer, it can instead extend "HTTPProvider" where some of the work is done for you.
 

core/constants.py

@@ -1,7 +1,7 @@
 from data.band import Band
 
 # General software
-SOFTWARE_NAME = "(S)pothole by M0TRT"
+SOFTWARE_NAME = "Spothole by M0TRT"
 SOFTWARE_VERSION = "0.1"
 
 # Special Interest Groups

providers/gma.py

@@ -1,3 +1,4 @@
+import logging
 from datetime import datetime, timedelta
 
 import pytz
@@ -27,24 +28,56 @@ class GMA(HTTPProvider):
             spot = Spot(source=self.name,
                         dx_call=source_spot["ACTIVATOR"].upper(),
                         de_call=source_spot["SPOTTER"].upper(),
-                        freq=float(source_spot["QRG"]) * 1000 if (source_spot["QRG"] != "") else None, # Seen GMA spots with no frequency
+                        freq=float(source_spot["QRG"]) * 1000 if (source_spot["QRG"] != "") else None,
-                        mode=source_spot["MODE"].upper() if "<>" not in source_spot["MODE"] else None, # Filter out some weird mode strings
+                        # Seen GMA spots with no frequency
+                        mode=source_spot["MODE"].upper() if "<>" not in source_spot["MODE"] else None,
+                        # Filter out some weird mode strings
                         comment=source_spot["TEXT"],
                         sig_refs=[source_spot["REF"]],
                         sig_refs_names=[source_spot["NAME"]],
-                        icon="person-hiking",
+                        time=datetime.strptime(source_spot["DATE"] + source_spot["TIME"], "%Y%m%d%H%M").replace(
-                        time=datetime.strptime(source_spot["DATE"] + source_spot["TIME"], "%Y%m%d%H%M").replace(tzinfo=pytz.UTC),
+                            tzinfo=pytz.UTC),
-                        latitude=float(source_spot["LAT"]) if (source_spot["LAT"] != "") else None, # Seen GMA spots with no lat/lon
+                        latitude=float(source_spot["LAT"]) if (source_spot["LAT"] != "") else None,
+                        # Seen GMA spots with no lat/lon
                         longitude=float(source_spot["LON"]) if (source_spot["LON"] != "") else None)
 
             # GMA doesn't give what programme (SIG) the reference is for until we separately look it up.
-            ref_response = self.REF_INFO_CACHE.get(self.REF_INFO_URL_ROOT + source_spot["REF"], headers=self.HTTP_HEADERS)
+            ref_response = self.REF_INFO_CACHE.get(self.REF_INFO_URL_ROOT + source_spot["REF"],
+                                                   headers=self.HTTP_HEADERS)
             # Sometimes this is blank, so handle that
             if ref_response.text is not None and ref_response.text != "":
                 ref_info = ref_response.json()
-                spot.sig = ref_info["reftype"]
+                # If this is POTA, SOTA or WWFF data we already have it through other means, so ignore. POTA and WWFF
+                # spots come through with reftype=POTA or reftype=WWFF. SOTA is harder to figure out because both SOTA
+                # and GMA summits come through with reftype=Summit, so we must check for the presence of a "sota" entry
+                # to determine if it's a SOTA summit.
+                if ref_info["reftype"] not in ["POTA", "WWFF"] and (ref_info["reftype"] is not "Summit" or ref_info["sota"] is ""):
+                    match ref_info["reftype"]:
+                        case "Summit":
+                            spot.sig = "GMA"
+                            spot.icon = "mountain"
+                        case "IOTA Island":
+                            spot.sig = "IOTA"
+                            spot.icon = "umbrella-beach"
+                        case "Lighthouse (ILLW)":
+                            spot.sig = "ILLW"
+                            spot.icon = "tower-observation"
+                        case "Lighthouse (ARLHS)":
+                            spot.sig = "ARLHS"
+                            spot.icon = "tower-observation"
+                        case "Castle":
+                            spot.sig = "WCA/COTA"
+                            spot.icon = "chess-rook"
+                        case "Mill":
+                            spot.sig = "MOTA"
+                            spot.icon = "fan"
+                        case _:
+                            logging.warn("GMA spot found with ref type " + ref_info[
+                                "reftype"] + ", developer needs to figure out an icon for this!")
+                            spot.sig = ref_info["reftype"]
+                            spot.icon = "person-hiking"
 
-            # Add to our list. Don't worry about de-duping, removing old spots etc. at this point; other code will do
+                    # Add to our list. Don't worry about de-duping, removing old spots etc. at this point; other code will do
-            # that for us.
+                    # that for us.
-            new_spots.append(spot)
+                    new_spots.append(spot)
         return new_spots

providers/hema.py

@@ -54,7 +54,7 @@ class HEMA(HTTPProvider):
                                 sig="HEMA",
                                 sig_refs=[spot_items[3].upper()],
                                 sig_refs_names=[spot_items[4]],
-                                icon="person-hiking",
+                                icon="mound",
                                 time=datetime.strptime(spot_items[0], "%d/%m/%Y %H:%M").replace(tzinfo=pytz.UTC),
                                 latitude=float(spot_items[7]),
                                 longitude=float(spot_items[8]))

views/webpage_about.tpl

@@ -1,11 +1,11 @@
 % rebase('webpage_base.tpl')
 
 <div id="info-container" class="mt-4">
-    <h3>About (S)pothole</h3>
+    <h3>About Spothole</h3>
-    <p>(S)pothole is a utility to aggregate "spots" from amateur radio DX clusters and xOTA spotting sites, and provide an open JSON API as well as a website to browse the data.</p>
+    <p>Spothole is a utility to aggregate "spots" from amateur radio DX clusters and xOTA spotting sites, and provide an open JSON API as well as a website to browse the data.</p>
-    <p>While there are several other web-based interfaces to DX clusters, and sites that aggregate spots from various outdoor activity programmes for amateur radio, (S)pothole differentiates itself by supporting a large number of data sources, and by being "API first" rather than just providing a web front-end. This allows other software to be built on top of it.</p>
+    <p>While there are several other web-based interfaces to DX clusters, and sites that aggregate spots from various outdoor activity programmes for amateur radio, Spothole differentiates itself by supporting a large number of data sources, and by being "API first" rather than just providing a web front-end. This allows other software to be built on top of it.</p>
     <p>The API is deliberately well-defined with an <a href="/apidocs/openapi.yml">OpenAPI specification</a> and auto-generated <a href="/apidocs">API documentation</a>. The API delivers spots in a consistent format regardless of the data source, freeing developers from needing to know how each individual data source presents its data.</p>
-    <p>(S)pothole itself is also open source, Public Domain licenced code that anyone can take and modify. <a href="https://git.ianrenton.com/ian/metaspot/">The source code is here</a>.</p>
+    <p>Spothole itself is also open source, Public Domain licenced code that anyone can take and modify. <a href="https://git.ianrenton.com/ian/metaspot/">The source code is here</a>. If you want to run your own copy of Spothole, or start modifying it for your own purposes, the README file there has all the details.</p>
     <p>Supported data sources include DX Clusters, the Reverse Beacon Network (RBN), the APRS Internet Service (APRS-IS), POTA, SOTA, WWFF, GMA, WWBOTA, HEMA, and Parks 'n' Peaks.</p>
     <p>The software was written by <a href="https://ianrenton.com">Ian Renton, MØTRT</a>.</p>
     <p><a href="/">&laquo; Back home</a></p>

views/webpage_base.tpl

@@ -9,8 +9,8 @@
     <meta name="apple-mobile-web-app-capable" content="yes">
     <meta name="apple-mobile-web-app-status-bar-style" content="white-translucent">
 
-    <meta property="og:title" content="(S)pothole"/>
+    <meta property="og:title" content="Spothole"/>
-    <meta property="twitter:title" content="(S)pothole"/>
+    <meta property="twitter:title" content="Spothole"/>
     <meta name="description" content="An Amateur Radio spotting tool bringing together DX clusters and outdoor programmes, providing a universal JSON API and web interface."/>
     <meta property="og:description" content="An Amateur Radio spotting tool bringing together DX clusters and outdoor programmes, providing a universal JSON API and web interface."/>
     <link rel="canonical" href="https://spothole.m0trt.radio/"/>
@@ -22,7 +22,7 @@
     <meta property="og:locale" content="en_GB"/>
     <meta property="og:type" content="website"/>
 
-    <title>(S)pothole</title>
+    <title>Spothole</title>
 
     <link rel="stylesheet" href="/css/style.css" type="text/css">
     <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.8/dist/css/bootstrap.min.css" rel="stylesheet"

webassets/apidocs/openapi.yml

@@ -1,10 +1,10 @@
 openapi: 3.0.4
 info:
-  title: (S)pothole API
+  title: Spothole API
   description: |-
-    (S)pothole is a utility to aggregate "spots" from amateur radio DX clusters and xOTA spotting sites, and provide an open JSON API as well as a website to browse the data.
+    Spothole is a utility to aggregate "spots" from amateur radio DX clusters and xOTA spotting sites, and provide an open JSON API as well as a website to browse the data.
     
-    While there are other web-based interfaces to DX clusters, and sites that aggregate spots from various outdoor activity programmes for amateur radio, (S)pothole differentiates itself by supporting a large number of data sources, and by being "API first" rather than just providing a web front-end. This allows other software to be built on top of it. (S)pothole itself is also open source, Public Domain licenced code that anyone can take and modify.
+    While there are other web-based interfaces to DX clusters, and sites that aggregate spots from various outdoor activity programmes for amateur radio, Spothole differentiates itself by supporting a large number of data sources, and by being "API first" rather than just providing a web front-end. This allows other software to be built on top of it. Spothole itself is also open source, Public Domain licenced code that anyone can take and modify.
   contact:
     email: ian@ianrenton.com
   license:

webassets/css/style.css

@@ -31,6 +31,7 @@ span.flag-wrapper {
 span.band-bullet {
     display: inline-block;
     cursor: default;
+    padding-right: 0.2em;
 }
 
 span.icon-wrapper {

webassets/js/code.js

@@ -96,7 +96,7 @@ function updateTable() {
         var mhz = Math.floor(s["freq"] / 1000000.0);
         var khz = Math.floor((s["freq"] - (mhz * 1000000.0)) / 1000.0);
         var hz = Math.floor(s["freq"] - (mhz * 1000000.0) - (khz * 1000.0));
-        var hz_string = (hz > 0) ? hz.toFixed(0) : "";
+        var hz_string = (hz > 0) ? hz.toFixed(0)[0] : "";
         var freq_string = `<span class='freq-mhz'>${mhz.toFixed(0)}</span><span class='freq-khz'>${khz.toFixed(0).padStart(3, '0')}</span><span class='freq-hz hideonmobile'>${hz_string}</span>`
 
         // Format the mode
@@ -114,6 +114,12 @@ function updateTable() {
             commentText = escapeHtml(s["comment"]);
         }
 
+        // Sig or fallback to source
+        var sigSourceText = s["source"];
+        if (s["sig"]) {
+            sigSourceText = s["sig"];
+        }
+
         // Format sig_refs
         var sig_refs = ""
         if (s["sig_refs"]) {