Skip to content

DDoS-Detector v54 – WGANGPHardeningAndResourceControl

Latest
Latest

Choose a tag to compare

View all tags
@BrenoFariasdaSilva BrenoFariasdaSilva released this 19 Jun 12:16
· 973 commits to main since this release
v54-WGANGPHardeningAndResourceControl
3be384d

This release focuses on hardening WGAN-GP execution, improving resource control, and fully standardizing dataset configuration and conversion.
It introduces dynamic num_workers computation, robust dataset conversion functions, improved configuration handling, and enhanced CLI/Telegram integration, strengthening stability, reproducibility, and maintainability across the DDoS-Detector framework.

These changes enhance the robustness of experimental pipelines while maintaining methodological consistency and improving maintainability across modules.

Key Features

WGAN-GP Execution and Resource Management

  • Added dynamic num_workers computation using (file_size_gb*3)/free_ram_gb.
  • Optimized execution to prevent memory exhaustion and runtime failures.
  • Improved reliability of training and generation pipelines.
  • Strengthened integration with Telegram reporting for resource utilization.
  • Standardized CLI boolean flags handling to prevent false sources.

Dataset Configuration and Conversion

  • Added datasets section in dataset_descriptor and dataset_converter configuration.
  • Implemented get_default_config, load_config_file, and initialize_defaults functions in dataset_converter.py.
  • Refactored dataset conversion to remove execution constants and use default configuration values.
  • Extracted and modularized functions for processing configured datasets, single input files, and input directories.
  • Extended support for configurable dataset converter variables in config.yaml and examples.

Architecture and Refactoring

  • Refactored imports, execution constants, and main function logic in dataset_converter.py.
  • Standardized configuration-driven dataset retrieval across modules.
  • Improved separation of concerns between:
    • dataset description,
    • conversion,
    • input path extraction,
    • batch processing.
  • Removed deprecated or unused configuration variables.
  • Minor refactors to improve maintainability and readability of dataset and Git commit scripts.

Tooling and Infrastructure

  • Updated Makefile to use virtual environment Python for pip upgrades.
  • Added git push functionality in commit scripts.
  • Improved minor try-except handling in dataset_descriptor.py.
  • Ensured default configurations define VERBOSE and prevent NameErrors in dataset conversion.

Documentation Updates

  • Updated config.yaml and examples with enhanced datasets and dataset converter sections.
  • Clarified dataset converter configuration variables.
  • Improved inline documentation for dataset initialization and batch conversion processes.

Impact

This release improves the stability, reproducibility, and resource efficiency of the DDoS-Detector framework.

By hardening WGAN-GP execution, standardizing dataset configuration and conversion, and introducing dynamic resource handling, the system becomes more predictable, easier to maintain, and resilient during large-scale data processing and experimental runs.

These changes strengthen the overall framework foundation without altering experimental methodology, ensuring safer and more reproducible machine learning pipelines.

Full Changelog:
v53-StackingDeterministicResultsAndPlots...v54-WGANGPHardeningAndResourceControl

Assets 2
Loading