Finding optimal hyperparameters of feedforward neural networks for solving differential equations using a genetic algorithm

Abstract

In this work, feedforward neural networks are used to solve 2D Laplace equation on rectangular domains. Optimal values of weights and biases of a network are recursively computed during the network training by minimizing a least-squares loss function using data at collocation points to approximate the true solution. The performance of the network largely depends on network architecture and model capability. In this work, an optimal set of hyperparameters is searched on various values of relative error. The genetic algorithm is used to find the optimal activation function, optimization algorithm, and weight initialization. In addition, we also searched for the optimal value of the number of hidden layers for a specific value of total parameters. Numerical results show that we can successfully obtain an optimal set of hyperparameters that is consistent across many values of relative error.