Epidemic spread in a population is traditionally modeled via compartmentalized models which represent the free evolution of disease in absence of any intervention policies. In addition, these models assume full observability of disease cases and do not account for under-reporting. We present a mathematical model, namely PolSIRD, which accounts for the under-reporting by introducing an observation mechanism. It also captures the effects of intervention policies on the disease spread parameters by leveraging intervention policy data along with the reported disease cases. Furthermore, we allow our recurrent model to learn the initial hidden state of all compartments end-to-end along with other parameters via gradient-based training. We apply our model to the spread of the recent global outbreak of COVID-19 in the United States, where our model outperforms the methods employed by the CDC in predicting the spread. We also provide counterfactual simulations from our model to analyze the effect of lifting the intervention policies prematurely and our model correctly predicts the second wave of the epidemic.