Modern convolutional neural networks (CNNs) require applying duplicate and redundant operations (such as alignment and matching) on different regions and pixels when processing three-dimensional (3D) reconstruction tasks. But different image areas and pixels are certainly not equally valuable for 3D reconstruction. In order to solve this problem, we propose a dynamic CNN structure for 3D reconstruction, which can dynamically modify the network based on the estimated impact of different regions and pixels on the 3D reconstruction task. The small gating branch learns which important areas or pixels need to be evaluated. The discrete gating decisions are trained with the Gumbel-Softmax trick, combined with a series of spatial and scale criteria. Our experiments on ShapeNet dataset shows that our method has higher accuracy than existing methods due to the better focus on important regions. Moreover, with an efficient CUDA implementation, our method achieves an improved inference speed on the most famous 3D reconstruction model Mesh R-CNN.