Pavement concrete undergoes fatigue load and freeze–thaw interaction in seasonal frozen regions, accelerating the deterioration rate of the concrete structure [1,2]. The performance and durability of pavement concrete will deteriorate due to microstructural damage occurring in the early stage of life [3,4]. As a result, the development and application of concrete pavement were restricted in seasonal frozen regions [5]. Pavement concrete is a porous material, and the strength of the concrete was affected directly by the internal pore structure. The pore structure deterioration was the attenuation mechanism of flexural strength [6–۸]. Omkar et al. compared the pore structure of concrete before and after fatigue load, and they concluded that an increase in pore volume fraction by approximately 10% resulted in a reduction in the compressive strength by approximately 50% [9]. Zhang et al. determined that the pore grading transferred from a smaller pore to a larger pore by studying the structure of concrete pores under the action of the freeze–thaw cycle [10]. Park et al. found that the long-term strength loss of concrete was caused by an increase in the total porosity and amounts of smaller pores in the material [11]. Some other literature showed that the loss of long-term strength is induced by an increase in porosity and an increased incidence of microcracking in cement paste [12,13]. On the other hand, some research has been conducted on the correlation between the macroscopic properties of concrete and its pore structure. Ozturk et al. established the relationship between concrete strength and porosity by regression analysis of a large number of experimental data [14]. Older et al. studied the strength and pore structure of concrete with different water-cement ratios and established a model of the relationship between strength and pore size distribution using a linear regression method [15]. Kumar et al. introduced the average pore size to establish the relationship between the concrete strength and pore structure based on the porosity [16]. Zhou et al. found that there was a good linear correlation between porosity parameters and strength [17]. Jin SS et al. established a multivariate model of the fractal dimension and strength of concrete through regression analysis [18]. Therefore, a considerable amount of research has been reported on concrete strength and pore structure. However, the current studies mainly focus on the compressive strength of the concrete and the effect of a single environmental factor on the concrete. The deterioration of the flexural strength of pavement concrete was more important than the compressive strength [19]. The pavement concrete in seasonal frozen regions is subject to load fatigue and freeze–thaw cycles, causing pore damage and attenuation of flexural strength [20]. Little studies involve the external environment that cannot reflect the true relationship between the pore structure and pavement concrete strength. This paper aims to determine the relationship between the pore structure of pavement concrete and its flexural strength under the interaction of fatigue load and the freeze–thaw cycle. Furthermore, the gray relational method was used to analyze the pore structure and flexural strength under this interaction. Finally, the model between the deterioration of the flexural strength of pavement concrete and the evolution of microstructures was established.