Merge pull request #43 from dimkroon/dev

Author: joaopa00

.github/workflows/updater.yml

@@ -1,4 +1,4 @@
-# This is a workflow to update the repository every day at 12:00PM
+# This is a workflow to update the repository every day at 22:00PM
 
 name: Updater
 
@@ -10,23 +10,98 @@ on:
 
 
 jobs:
-  cron:
+  update-epg-data:
     runs-on: ubuntu-latest
 
     steps:
-    - uses: actions/checkout@v3
-      
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        pip install pytz
-        sudo apt-get install -y xmltv
-        sudo apt-get install -y libmojolicious-perl
- 
-    - name: update 
-      working-directory: scripts
-      run: |
+      - name: Checkout code branch
+        uses: actions/checkout@v6
+
+      - name: Checkout data branch
+        uses: actions/checkout@v6
+        with:
+          ref: master
+          path: databranch
+
+      - name: Print folder structure
+        run: ls -R ./
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install pytz
+          sudo apt-get install -y xmltv
+          sudo apt-get install -y libmojolicious-perl
+
+      - name: Copy raw files from data branch
+        run: cp -rf databranch/raw ./
+        continue-on-error: true
+
+      - name: Print folder structure
+        run: ls -R ./
+
+      - name: Update
+        working-directory: scripts
+        run: python3 update_all_tv_guides.py
+
+      - name: Print folder structure
+        run: ls -R ./
+
+      - name: Store raw data
+        uses: actions/upload-artifact@v6
+        with:
+          name: raw-data
+          path: raw/
+          retention-days: 7
+
+      - name: Store EPG data
+        uses: actions/upload-artifact@v6
+        with:
+          name: epg-data
+          path: data/
+          retention-days: 2
+
+
+  push-epg-data:
+    needs: update-epg-data
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+
+    steps:
+      - name: Checkout data branch
+        uses: actions/checkout@v6
+        with:
+          ref: master
+
+      - name: Print folder contents after checkout
+        run: ls -A ./
+
+      - name: Clean existing data
+        run: rm -rf ./*
+
+      - name: Print folder contents after cleanup
+        run: ls -A ./
+
+      - name: Download stored EPG data
+        uses: actions/download-artifact@v7
+        with:
+          name: epg-data
+
+      - name: Download stored raw files
+        uses: actions/download-artifact@v7
+        with:
+          name: raw-data
+          path: raw
+
+      - name: Print folder structure
+        run: ls -R ./
+
+      - name: Force-push new data
+        run: |
+          now=$(date +"%d/%m/%Y %H:%M:%S-%Z")
           git config user.name 'GitHub Actions'
           git config user.email 'github@noreply.github.com'
-          bash update_all_tv_guides.sh
-      shell: bash
+          git add --all
+          git commit --amend -m "Auto update TV guides ($now)"
+          git push -f origin master

.gitignore

@@ -33,3 +33,11 @@ Temporary Items
 __pycache__/
 *.py[cod]
 *$py.class
+
+
+/raw/
+cache/
+.cache/
+.local/
+.selected_editor
+.bash_history

README.md

@@ -9,20 +9,33 @@ This repository keeps XMLTV TV guides up to date in order to be used with the Li
 
 ## Developers notes
 
-The `update_all_tv_guides.sh` can be used to automatically update TV guides files (`xxxxx.xml` files).
-The script automatically create a commit with the latest TV guides and it will push the modification on this GitHub repository.
-This script is executed every night with a cron task but you can trigger it manually on your own computer.
-
-## Fonctionnement (only in french ATM)
-
-* Si on est le jour J (par exemple le 19/06/2020)
-* Toutes les nuits, le script bash `update_all_tv_guides.sh` est exécuté
-* Ce que fait le script bash (`update_all_tv_guides.sh`)
-    * Pull du dépôt GitHub
-    * Suppression des fichiers XMLTV de la racine
-    * Pour chaque pays, récupération du programme TV de la journée J+2 (par exemple le 21/06/2020) qu'on place dans le dossier `raw`
-    * Appel du petit script Python (`post_treatment.py`)
-        * Suppression, dans le dossier `raw`, des programmes TV trop anciens
-        * Fusion, pour chaque pays, des programmes TV de chaque jour présents dans le dossier `raw`
-        * Post traitement des programmes TV (découpage par jour, avec heure locale ou UTC, ...)
-    * Push sur le dépôt GitHub des modifications
+Contrary to what the name suggests, the `master` branch is not the default 
+branch in this repo. It contains no code, just EPG data files. The default 
+branch is `dev`, which contains all code and no data. All PR's should target 
+the `dev` branch.
+
+The `update_all_tv_guides.py` script can be used to automatically update TV 
+guides files (`xxxxx.xml` files). This script is executed every night from a 
+GitHub workflow that is triggered by a cron task, but you can run it manually 
+on your own computer. The workflow automatically create a commit with the 
+latest TV guides and it will push the modifications to the master branch on 
+this GitHub repository.
+
+## How it works
+
+* Every night at 22:00 the workflow runs.
+* Pull the `dev` branch from the GitHub repo.
+* Pull the `master` branch from the GitHub repo and copy the folder with raw 
+  files.
+* Python script `update_all_tv_guides.py` is executed
+* What the script does (`update_all_tv_guides.py`):
+  * Delete XMLTV files from the root directory
+  * Deletes outdated files from the `raw` folder
+  * For each country, always retrieve the file with TV programmes for tomorrow 
+    and place it in the `raw` folder.
+  * Determine how many days of EPG should be available for each country and 
+    retrieve the programmes for each day for which no raw file is yet available. 
+  * Calls the small Python script (`post_treatment.py`)
+  * Merges the TV programmes for each day in the `raw` folder for each country
+  * Post-processes the TV programmes (divides them by day, with local time or UTC, etc.)
+* Pushes xmltv files and raw files to the master branch of the GitHub repository