Parameters

In this step, the hyperparameters, model architecture, and system settings can be configured for the training process. These parameters influence training stability, convergence behavior, computational performance, and resource usage.

Why this matters

• Convergence: proper configuration of the optimizer and learning-rate scheduler help the model converge reliably and reduce the risk of unstable training.
• Resource Management: parameters such as Batch Size and Num Workers control GPU memory usage and data loading speeds.
• Model Architecture: selecting a model architecture that matches the task complexity helps balance learning capacity and generalization.

DataLoader Config

This sections describes the hyperparameters that control data samples loading and how they are delivered to the model during training and validation. Efficient data loading can lower the duration of the training process.

• Batch Size: Specifies the number of samples processed by the model in a single training/validation step. Larger batch sizes can improve the GPU utilization and stabilize the training of the model. However, they also increase GPU memory usage.

  💡

  If out-of-memory (OOM) errors occur, reducing the batch size is usually the first mitigation step.

• Shuffle: When enabled, training samples are randomly shuffled at the beginning of each epoch. This method improves the model generalization.
• Num Workers: Determines the number of parallel worker processes used to load data. Increasing this value can reduce data loading latency, but the optimal setting depends on the available CPU cores, storage speed, and memory bandwidth.
• Prefetch Factor: Controls how many batches each worker loads in advance, Prefetching helps ensure that the data is available as soon as possible when requested by the trainer. Thus, it is reducing the idle time caused by data loading delays.

Training Strategies

Training strategies define the high-level components that control how the model is trained, optimized and evaluated.

Training Config

Defines the training loop behavior and specific loss functions.

• Object Detection Training Config

  • Number of epochs: Total number of training cycles through the entire dataset. More epochs can lead to better performance but take more time. (Default: 10)
  • Early stopping patience: Number of epochs to wait for improvement in the monitored metric before stopping training. Helps prevent overfitting. (Default: 30)
  • Gradient clipping: A technique used to prevent gradients from becoming too large, which can make training unstable. Set to 0 to disable. (Default: 0)
  • Use Automatic Mixed Precision (AMP): Whether to use half-precision floating-point numbers during training. This can significantly speed up training and reduce memory usage on modern GPUs. (Default: false)

• Anomaly Detection Training Config

  • Number of epochs: Total number of training cycles through the entire dataset. More epochs can lead to better performance but take more time. (Default: 10)
  • Early stopping patience: Number of epochs to wait for improvement in the monitored metric before stopping training. Helps prevent overfitting. (Default: 30)
  • Gradient clipping: A technique used to prevent gradients from becoming too large, which can make training unstable. Set to 0 to disable. (Default: 0)
  • Use Automatic Mixed Precision (AMP): Whether to use half-precision floating-point numbers during training. This can significantly speed up training and reduce memory usage on modern GPUs. (Default: false)

Optimizer Config

Choose the algorithm used to update model weights.

• Adam: A popular optimization algorithm that adapts the learning rate for each parameter.

  • Learning rate: A fundamental parameter that controls how much the model weights are adjusted in response to the estimated error. (Default: 0.001)
  • Beta coefficients: Coefficients used for computing running averages of gradient and its square. (Default: [0.9, 0.999])
  • Epsilon: A small value added to the denominator to improve numerical stability and avoid division by zero. (Default: 1e-8)
  • Weight decay (L2 penalty): A regularization technique that applies a small penalty to large weights, helping to prevent overfitting. (Default: 0)

• SGD: Stochastic Gradient Descent optimizer, a simple yet effective optimization algorithm.

  • Learning rate: A fundamental parameter that controls how much the model weights are adjusted in response to the estimated error. (Default: 0.001)
  • Momentum: Helps accelerate SGD in the relevant direction and dampens oscillations. (Default: 0)
  • Weight decay (L2 penalty): A regularization technique that applies a small penalty to large weights, helping to prevent overfitting. (Default: 0)
  • Dampening: A value used to reduce the effect of momentum. (Default: 0)
  • Use Nesterov momentum: Whether to use Nesterov momentum, which is a variant of the momentum method that often leads to faster convergence. (Default: false)

Scheduler Config

Defines the policy for adjusting the learning rate during training.

• StepLR: Adjusts the learning rate by a fixed factor after a set number of steps.

  • Update interval: Determines if the learning rate should be updated after every epoch or after every step. (Default: epoch)
  • Update frequency: How often the learning rate should be updated. (Default: 1)
  • Metric to monitor: The performance metric that the scheduler will monitor. (Default: val/loss)
  • Step size: Number of epochs (or steps) after which the learning rate will be reduced. (Default: 30)
  • Gamma (reduction factor): The factor by which the learning rate will be multiplied after each step size. (Default: 0.1)

• MultiStepLR: Adjusts the learning rate by a fixed factor at specific epoch/step milestones.

  • Update interval: Determines if the learning rate should be updated after every epoch or after every step. (Default: epoch)
  • Update frequency: How often the learning rate should be updated. (Default: 1)
  • Metric to monitor: The performance metric that the scheduler will monitor. (Default: val/loss)
  • Milestones: A list of epoch (or step) numbers where the learning rate should be reduced.
  • Gamma (reduction factor): The factor by which the learning rate will be multiplied at each milestone. (Default: 0.1)

• Cosine: Adjusts the learning rate following a cosine curve, providing a smooth decay from the initial value to a minimum.

  • Update interval: Determines if the learning rate should be updated after every epoch or after every step. (Default: epoch)
  • Update frequency: How often the learning rate should be updated. (Default: 1)
  • Metric to monitor: The performance metric that the scheduler will monitor. (Default: val/loss)
  • Maximum iterations: The number of iterations (usually epochs) for one full cycle of the cosine annealing. (Default: 10)
  • Minimum learning rate: The lowest value that the learning rate can reach during the cosine annealing. (Default: 0)
  • Last epoch index: The index of the last epoch. Set to -1 to start from the beginning. (Default: -1)

Evaluation Config

Specifies metrics and protocols for validating model performance.

• Object Detection Evaluation Config

  • Main evaluation metric: The primary metric used to evaluate the model (e.g., mAP50).
  • Evaluation metrics: List of metrics to be calculated during evaluation (e.g., loss, mAP50, mAP75, mAP90).

• Anomaly Detection Evaluation Config

  • Main evaluation metric: The primary metric used to evaluate the model (e.g., AUROC, F1Max).
  • Evaluation metrics: List of metrics to be calculated during evaluation (e.g., AUROC, F1Max).
  • Detection threshold: Optional fixed threshold for binary decision.
  • Image-level threshold: The threshold used to classify entire images as anomalous. (Default: 0.5)
  • Pixel-level threshold: The threshold used to identify anomalous pixels. (Default: 0.5)

Nn Model Config

Selects the neural network architecture.

• Faster R-CNN (Object Detection)

  • Use pretrained model: Whether to load weights from a model previously trained on the COCO dataset. (Default: true)
  • Backbone architecture: The base neural network architecture (e.g., resnet50, mobilenet). ResNet50 is more accurate, MobileNet is faster.
  • Trainable backbone layers: The number of layers in the backbone that will be updated during training. (Default: 3)
  • Score threshold: Confidence threshold for detections. (Default: 0.05)
  • NMS threshold: Non-Maximum Suppression threshold. (Default: 0.5)
  • Max detections per image: The maximum number of objects the model can detect in a single image. (Default: 100)

• FastFlow (Anomaly Detection)

  • Use pretrained model: Whether to load weights from a model previously trained on a large dataset. (Default: true)
  • Backbone architecture: The base neural network architecture (e.g., resnet18, wide_resnet50_2). ( Default: resnet18)
  • Flow steps: The number of transformation steps in the normalizing flow. (Default: 8)
  • Hidden ratio: The ratio of hidden layer size to input size in the flow transformations. (Default: 1)

• PatchCore (Anomaly Detection)

  • Use pretrained model: Whether to load weights from a model previously trained on a large dataset. (Default: true)
  • Backbone architecture: The base neural network architecture (e.g., resnet18, wide_resnet50_2). ( Default: resnet18)
  • Subsample fraction: The fraction of the memory bank to keep after subsampling. (Default: 0.1)
  • Number of neighbors: Number of neighbors.

How to configure

1. Tune Batch Size: Set as high as your GPU memory allows for efficiency, or lower it for stability.
2. Verify configurations: Ensure the selected configurations (Training, Model, Evaluation) are compatible with your dataset type (e.g., Object Detection vs. Classification).
3. Review Optimizer: "Adam" is a good default for many tasks, but specific models may require others.