On Grimdark, I feel like it works when the story is committed to it, such as the Vaults of Terra or Inquisition War series. It also works when used as satire, such as in the Ciaphas Cain books. However, I find a lot of the basic Bolter Porn books fall in an uncomfortable middle, where the space marines or IG are too clearly heroes (as opposed to protagonists), and the setting seems less to be saying "the Imperium is horrible and this makes its problems even worse" than "the Imperium is horrible, and is justified".
As for the background discussion, things may be changing. WH40k's background has been declining in quality since 5th edition, at least in the codices. BL has become wildly uneven over the course of the Horus Heresy series, and I find myself less likely to pick up a new BL book unless I have heard a lot of recommendations for it. The old FFG RPG depth of background material will likely never be surpassed at this point.
I expect The Old World will be similarly disappointing. Even BL was losing the plot on WHF after the Storm of Chaos; WHT books require a specific 'flavor' to work for me, and I wasn't finding it in the End Times books or their weird HH knockoff books about the mythohistorical events that shaped the Old World. The books I tend to go back to are the William King Gotrek books, Kim Newman booksand the CL Werner books.
In light of this, the competition is getting closer. Mantic may have dropped the ball when they had the chance to score big by keeping their setting generic so it wouldn't scare if the former WHFB players (or just because they lacked vision), but now they have an RPG and a series of novels coming out set in their Pannithor world, including one by CL Werner. Their background material also expanded a bit with the Armada game. They were late to the background game, but soon they should have enough to keep lore-loving gamers interested.
Their Firefight universe got lucky in that Dreadball gave a whole third dimension to their background, Verhoeven-style satire, making it a lot more fun than Deadzone/Warpath's blander Late Stage Capitalism in Space take. They are putting out background articles every Thursday, and the setting is a fun place to read about.
Frostgrave has some short stories and novels out, but they still feel terribly generic. There isn't any attempt that I can see to make the background fun beyond each individual writer's storytelling ability. I haven't looked into Stargrave at all, but unless someone tells me otherwise I'll assume it's the same.
As for other potential contenders...
Maelstrom's Edge really tried to develop their fluff. They gave it some good effort. Unfortunately, the central feature of their background is grimdark, and it taints every story in the setting, making even the most uplifting story kind of depressing. It's just not a fun place to spend imagination time.
AT-43 had some brilliant background. Too bad that game's dead forever.
Mutant Chronicles/Warzone had a lot of fun background. Too bad that game is dead forever.
Confrontation had a lot of background. That game is dead forever.
Warmachine/Iron Kingdoms had a lot of background that appealed to a lot of somebodies, apparently. Too bad that game is dead forever any day now.
Conquest: Last Argument of Kings has a background book that costs damnnear $50. It could have the greatest background of all time, but we'll never know it just from buying the starter set. They leaned hardcore into the crunchy tournament scene at the expense of casuals and narrative players, and the second they lose "balance" there will be no die hard fans there to catch them.
Beyond the Gates of Antares had a fairly simple background conceit, and some surprisingly engaging fiction. The background was a fun place to spend time. Then Warlord Games pulled a Warlord Games. Too bad the game is Warlord Gamesed forever.