In the traditional system identification techniques, a priori model structure is widely assumed to be available and the unknown parameters of the assumed model structure are estimated by maximizing the adherence of the assumed model structure to the experimental data. However, selecting the model structure can be problematic, sometimes leading to overfitting. Recent developments in the regularization based system identification methods, which are inspired from the fields of machine learning and statistics, showed promising results in avoiding the overfitting in non-parametric identification. For instance, the objective function of the atomic-norm regularized identification problem consists of two terms: one promotes the adherence, while the other promotes a low-order model. However, the regularization approaches might induce biased estimations depending on the regularizer function which result in a decrease in the estimation performance. Hence, the aim of this thesis is to investigate the debiasing strategies for the regularized identification techniques. This thesis extends the novel atomic-norm regularized identification problem, by using the recent developments in the sparse estimations for high-dimensional statistics to improve the bias property of the estimation, while increasing the quality of identification. For this purpose, the adaptive selection of the penalization parameters via the adaptive lasso and non-convex penalties, and the filtering of the noise variables via the bootstrapping-based stability selection techniques are proved to be useful for the optimization over the uniformly discretized and fixed atomic set. The thesis also investigates the effect of the adaptively generated atomic set, which is iteratively constructed by controlling the atom selection process via prefiltering of the random-uniformly generated atomic set using the baseline estimator and its optimality condition.