Skip to content

rodekruis/IBF_TYPHOON_DATA_PIPELINE

Repository files navigation

Table of contents

Table of Contents
  1. Typhoon Impact forecasting model
  2. Installation
  3. Running pipeline With Docker
  4. Acknowledgments

Typhoon Impact forecasting model

This tool was developed as a trigger mechanism for the typhoon Early action protocol of the Philippines Red Cross FbF project. The model will predict the potential damage of a typhoon before landfall, and the prediction will be percentage of completely damaged houses per municipality. The tool is available under the GPL license

To run the pipeline, you need access to an Data.zip, and credentiials for 510 Datalake and FTP server. If you or your organization is interested in using the pipeline, please contact 510 Global to obtain the credentials. You will receive a file either called secrets or .env, which you need to place in the top-level directory.

The main use of this data pipeline is to update the status of Typhoon IBF portal based on the latest ECMWF forecast. The status of Triggered/Not Triggred is defined based on EAP trigger value. Currently this is based on Average impact. This value can be updated the setting.py file at this line Defult value is Average, other possible values are 50,70 and 90 , which is the percentage of ensamble members passing the treshold.

Installation

  1. Clone the repo
  2. Fill in necessary credentials:
    • If using secret.py, change /IBF-Typhoon-model/src/typhoonmodel/utility_fun/secrets.py.template to secrets.py and fill in credentials.
    • If using .env, change /IBF-Typhoon-model/.env.template to .env and fill in credentials.
  3. Install Docker-compose follow this link

Running the model with jupyter notebook

Step 1: Create Conda Environment

Create a new conda environment named tyworkflow with all required packages by running:

conda create -n tyworkflow python=3.11 ipykernel pandas numpy geopandas scikit-learn xgboost rasterstats xarray rasterio fiona shapely cartopy dask earthpy pycountry contextily imageio h5py numba pathos cftime cfgrib netCDF4 eccodes statsmodels requests pybufrkit pint feedparser lxml beautifulsoup4 tqdm azure-storage-file azure-identity azure-storage-blob azure-storage-file azure-storage-file-datalake python-dotenv -c conda-forge

Step 2: Install Visual Studio Code (or any Jupyter-compatible IDE)

Make sure you have VS Code, JupyterLab, or another IDE that supports Jupyter notebooks.

Step 3: Open the Project and Run Notebook

  • Open the project folder in Visual Studio Code.
  • Open and run the pipeline.ipynb notebook within the tyworkflow environment.

Running With Docker on local machine

You will need to have docker and docker-compose installed.

Build Container

To build and run the image, ensure you are in the top-level directory and execute:

docker-compose up --build


When you are finished the run

docker-compose down

to remove any docker container(s).

Running with Docker on Azure logicapp

Follow the instraction here there is also a workflow in github action to update docker image in azure registery, which will be the image used in logic app.

Acknowledgments

About

No description or website provided.

Topics

Resources

License

Stars

Watchers

Forks

Packages

 
 
 

Contributors