We have experimentally observed the eigenmode splitting due to coupling of the evanescent defect modes in the photonic band gap structures. The splitting was well explained with a theory based on the classical wave analog of the tight-binding (TB) formalism in solid state physics. The experimental results were used to extract the TB parameters. A new type of waveguiding mechanism in the photonic structures is demonstrated. Photons propagate through strongly localized defect cavities due to coupling between adjacent cavity modes. High transmission of the electromagnetic waves, nearly 100 percent, is observed for various waveguide structures even if the cavities are placed along an arbitrarily shaped path. The dispersion relation of the waveguiding band is obtained from transmission-phase measurements, and this relation is well explained within the tight-binding photon picture. Various applications of the coupled cavity structures were proposed and demonstrated.