A continuum theory of elasticity based on the concept of interface free energy density is proposed to account for the effect of incoherent interfaces in nano-phase reinforced composites. With the help of the lattice model, the corresponding interface energy density is formulated in terms of the surface free energy densities of two bulk materials forming interfaces, the lattice relaxation parameters due to the spontaneous surface relaxation and lattice misfit parameters yielded by interface incoherency, while the stress jump at interfaces is formulated with an interface-induced traction as a function of interface free energy density. Compared with existing theories, the interface elastic constants difficult to determine are no longer introduced, and all the parameters involved in the present theory have definite physical meanings and can be easily determined. The coupling effects of characteristic size and interface structure in nanoparticle-reinforced composites are further analyzed with the present theory. It is found that both the decrease of nanoparticle size and the increase of interface incoherence will lead to the decrease of interface fracture toughness and increase of effective bulk and shear moduli of nanocomposites. All these results predicted by the present theory are consistent well with those obtained by previous experiments and computations, which further indicate that the present theory can effectively predict the mechanical properties of nanomaterials with complex interfaces, such as nano-phase reinforced composites and nano-scale metal multilayer composites.