Formatting Gratitude

“Don’t take the job,” he said.

Almost thirteen years have passed since my friend and mentor shared these words. He was the only dissenting voice among half a dozen professionals and friends whose opinions I sought. And it’s not those four words, but the rationale behind them, that has stuck with me the most.

Flashback to the brisk autumn of 2014. I had earned PhD candidacy the year before, passed comprehensive exams the previous semester, and my son was entering his sixth month of life on Earth. It was an exciting time—and it was about to get a lot more exciting.

A new role had just opened up at NASA’s Earth Observatory. They needed a lead data visualizer. As it would turn out, I knew just the guy.

A Crossroads

Applying for a new job is always exciting. But I wasn’t just applying for a job. I was staring down a major career decision. One that would have a profound effect on the remainder of my life: to stay in academia or leave.

A number of factors contributed to my choice to do the latter. And it was among the better decisions I have ever made. But it was not an easy choice. Over the course of several weeks I met with just about every professor, advisor, friend, and industry professional in my life whose advice and thoughts I deeply admire.

Their thoughts were generally unanimous. “This is a phenomenal opportunity and a fantastic match.”

But one friend and mentor felt otherwise.

“Don’t take the job,” he said as we ate lunch (Qdoba, I think). “You will end up dealing with NetCDFs, HDFs, and some really weird data formats. It will be a never-ending headache.”

This person is a highly cited, well-respected professor. He works closely with those at NASA all the time. He knew exactly what he was talking about. It didn’t take long in my role as lead visualizer to tell you that he was 100% correct.

Over nearly a decade at NASA I dealt with (tens of?) thousands of NetCDF and HDF files. I have had to process binary files in whatever pet project of a format a scientist felt like creating that week.

And I am grateful for the experience.

It pains me to consider what my life would have been like had I turned down a career opportunity to avoid unfamiliar data formats. It also pains me to consider how common this sentiment is, and the lengths folks are willing to go to avoid a data format. So much so that turning down an incredible career opportunity over a data format could seem reasonable.

Again—my friend was not wrong about encountering some downright weird data on a regular basis. Or the headache it could be.

But he was wrong to assume this would be an insurmountable, and unrewarding, problem. He was also wrong about the headache—it was only minor and temporary.

That’s where this story gets less personal.

Embrace difficulty

In a lot of ways, I think my friend’s concern is shared by many in the geospatial community. A large swath of people have such a dislike of NetCDF/HDF-type files they see these files as a challenge best ignored. To many, NetCDFs are not files—they’re problems; problems for someone else to solve, and problems that would be unrewarding to deal with on one’s own.

The first few months on the job were challenging. A well-formatted NetCDF file following standard conventions “just works.” But files in research environments get created quickly—often as intermediary steps in a much larger process—and such conventions go ignored. If data wrangling were a game, Earth science will introduce you to the final boss fairly quickly.

But, just like Dark Souls, with the challenge comes the reward.

Following some tutorials, signing up for a class (your job will probably pay for it!), or spending a few weekends tinkering with Python is all it really takes. And the payoff will last for decades. Literally!

The problem isn’t you

There are a lot of complaints about NetCDF and HDF as formats. Many of them are valid.

But what I have noticed is that much of the difficulty is because the software we use most doesn’t read particular files. That’s a shortcoming of that software, not NetCDF files.

“That wouldn’t happen if the files were formatted properly!” critics of NetCDF will say. And they’re right! But again, this is not a problem with NetCDF. A poorly or incorrectly created GeoTIFF will fail to be read properly, too.

While the problem is not with NetCDF, this is not to say the problem is with you. The solution, however, is firmly in your court.

If your software of choice fails to read a NetCDF, converting the data to something your software will read is a lot easier than you may realize. To find out though, you have to give it a try.

…but the solution is

Anytime I share a map produced from a well known NetCDF source, a link to data that is a NetCDF, or simply hint to the existence of NetCDFs, the responses are almost immediately critical. This critique is largely targeted at NetCDF itself. But it has been personal (leading to the only time I’ve ever blocked someone on social media!).

The vast majority of the emails, direct messages, and replies I’ve received about NetCDF have been complaints. “_____ software doesn’t read NetCDFs,” “I can’t believe this data is only available in NetCDF!”, “I need the data in GeoTIFF/Shapefile/CSV…”

After years and years of these replies, do you know what type of message I have never seen? Not even once?

One that might go something like this:

“I’ve downloaded a NetCDF from ______. My usual software wouldn’t read it, so I tried following this tutorial. I am getting stuck on step 3. The error I get is ______. Could you help? What’s going wrong?”

Not. Once.

My suspicion is that those who do put some skin in the game and give it a try immediately see a solution is possible. They realize it wasn’t nearly the ordeal they were led to believe and move on with a new skill in hand.

And today the barrier to entry is even lower. Just about any AI will one-shot your file conversion nearly 100% of the time.

Tricky file formats were not an insurmountable problem in 2014. They are a complete nonissue in 2026.

And I am forever grateful for the opportunity to learn these formats, solve the problems commonly associated with them, and learn that a little effort goes a long way.

This post was written several years ago. It collected dust as a draft, and was just recently edited and brought to the light of day.

---