Every generation tends to reevaluate and response against the aesthetics of the previous generation, though in the case of Hollywood, you have the situation of a very new art form with cinematographers who started in the late 1920's and retired in the early 1970's, so even among the same generation, there was stylistic trends, some in parallel with audience's changing tastes. Same happened in art, theater, architecture, fashion, etc.
The 1930's were an era of studio glamour that began in the late 1920's, a sort of soft, gauzy style -- softer lights, softer lenses, diffusion, etc. Back then you'd see a whole movie shot through nets and glass diffusion.
By the 1940's there was a shift to greater sharpness, except for close-up work, which still tended to be diffused and romantic, just less so than in the 1930's (and less so for 3-strip Technicolor movies, which generally aimed for greater sharpness to offset any problems in aligning three color records in the print, plus the fact that shooting almost wide-open for interiors tended to soften the image anyway, so there was less need for lens diffusion.) I think some people embraced harder lighting and sharper lenses (like Gregg Toland) out of a belief that this was more "realistic" (certainly it was more dramatic) while glamour and fashion also shifted towards sharper, clearer images -- and I doubt that this was because realism was their priority. It may have simply been tastes changing after a decade or two of soft images, maybe the ad world thought that they looked "old-fashioned" and that "modern" glamour and romance was bolder, clearer, more in your face so to speak, like a Kodachrome slide was. Women were no longer being portrayed as soft, demure, mysterious creatures or delicate, but strong, aggressive, statuesque, confident, etc. So the stronger, sharper lighting was a new way of glamorizing actors, not making them more realistic, though for some dramas, that was also a motivating factor, though today we would consider the harder lighting to be less realistic, not more. But that generation defined "realism" differently, they thought if it as showing the world as it was, more clearly, they didn't think of it so much as mimicking natural light effects except for special sequences in the movie. So for them, it was more "realistic" to light a tough guy with hard cross-lighting to bring out his chiseled rough features rather than use soft "romantic" lighting on him. Today, we'd light a tough guy in soft light if he were standing in a room naturally lit with soft light, though when given a chance to "make up" the light, like a night exterior, we may opt for harder lighting.