| Description |
Structured prediction is the machine learning task of predicting a structured output given an input. For these problems constraints among output variables play significant roles in both the learning and prediction phases. The goal of this dissertation is to improve a structured-prediction model's performance by understanding constraints in the problem. More specifically, I will focus on learning useful constraints from data and reduce inference time at the presence of complicated constraints. I use neural networks to learn constraints from the training data of structured prediction problems. I frame the problem as that of training shallow rectifier networks to identify valid structures or substructures. The trained rectifier networks are not readily useful in structured prediction problems. To convert the trained networks to a form which can be directly used in various structured prediction inference algorithms, I derive a result on the expressiveness of rectifier networks and use it to convert the trained networks to systems of linear inequality constraints. I empirically verify the effectiveness of the learned constraints by performing experiments in the information extraction domain. I show that the learned linear inequality constraints are very effective in improving the prediction performance of an existing structured prediction model, and they can even be used in the training phase to obtain a better classifier. When complicated constraints exist, inference in a structured prediction problem is often hard. I consider the problem of reducing the inference time of a trained black-box classifier without losing accuracy. To do so, I train a speedup classifier that learns to imitate a black-box classifier under the learning to search approach. As the structured classifier predicts more examples, the speedup classifier will operate as a learned heuristic to guide search to favorable regions of the output space. Evaluations on the task of entity and relation extraction show that the speedup classifier outperforms even greedy search in terms of speed without loss of accuracy of the original black-box classifier. In the last part of the dissertation, I derive a way of constructing the decision boundary for deep neural networks with piecewise linear activations. This construction is an extension of the expressiveness results on shallow rectifier networks. The decision boundaries of such networks are composed of a system of hyperplanes. I identify these decision hyperplanes and describe a way of combining them to construct the decision boundaries of the deep neural networks. iv |
