The Richmond Fed says manufacturers in those regions could bear the brunt of the consequences, with net job losses.