I always use transparent foil that is usually used for overhead projectors (you know when you make presentations with permanent color markers on the foil to project it.
They are so easily cut to any shapes, transparent like a real glass, thin enough and in the same time thick enough to imitate glass window, could be glued by pvc or acrylic glues, and once fitted into the frame- indistinguishable from the real glass. Such foil windows could be even curved to imitate curved glass or windows without problems. In addition, if you have more small windows in a row, you could cut appropriate strips of the foil, glue it from the rear side of the multiple frames and thus simplify the procedure.
I always prefer transparent or semi-transparent materials for windows which give the hint if the inside, giving the depth to the models (if possible by construction) than opaque materials that just imitate glass.