Ceph Principal Architect, Red Hat
Sage Weil started seeing the need for an open source storage alternative to proprietary data storage solutions while at college. “I co-founded a web hosting company with a bunch of other students called DreamHost. We were storing and hosting a bajillion websites, we had lots of data, and at some point we switched to using centralised storage: we started buying netapps and it was the greatest thing ever to have all our storage in one place. A couple of years down the line, however, and we'd grown to the point where we had about a hundred netapps. We were writing software to manage moving data around. It had all become nightmarishly complicated, not to mention expensive,” he laughs. “The problem was that, at that point in time, there were no open source storage systems that were scalable. You had to either cobble something together with a bunch of machines or go buy some expensive enterprise solution.”
Around this time, Weil started Grad School at the University of California: Santa Cruz, where he ended up in the storage research group - a group which also collaborated with some of the US’ most prestigious national labs, like Los Alamos, on scalable, petabyte-scale file systems. “The Ceph architecture came out of that research group,” he explains. “There were all sorts of interesting ideas we had back then about data distribution, consistency, and so on that we assembled into a working prototype in 2007.”
By 2011, Weil and a few colleagues started Inktank, the lead development contributor and financial backer for Ceph. He reflects that “I was pretty naive back then. I thought you just built something, open sourced it, and people would just start appearing to develop it, fix bugs, etc. and that's not how it works.” As Ceph reached a level of maturity where Sage and the Inktank team felt that “people would actually be willing to use it,” he explains that “around that time we started bringing on community managers and people who actually knew about cultivating an ecosystem, and we started talking about creating a foundation for Ceph. But then we got bought by Red Hat and that can was kicked down the road a bit.”
Weil - along with Red Hat, and some of the biggest names in the open source tech space and beyond - officially launched the Ceph Foundation in 2018, under the Linux Foundation’s umbrella. In those intervening years (and since) Weil reflects that the sheer size, depth, and diversity that’s developed in the Ceph community is staggering.
“We never really know how big our community is or how many people are using Ceph. We have to go on data like how many bug reports we get, how many patches we get, how active our user list is, etc. and those metrics definitely give us a glimpse into just how big and varied the Ceph user base is and how deep its penetration is into certain spaces.
“We talk to a lot of users that love Ceph because it allows them to do things that they simply couldn't do if they were forced into an expensive proprietary solution.
“One conversation that stands out to me was with a researcher at a University in Canada that does research for a cancer institute. They told me that Ceph is what allows them to do the research that they do, because they wouldn't have the funding to support the massive amounts of data storage that their research requires if they didn't have an open source, low-cost solution.
“In fact, Ceph is widely used throughout a lot of the scientific community. We have a very robust relationship with the group at CERN, and we know they collaborate with scientific communities around the world. A lot of them are deploying Ceph to do all sorts of different things, like these scientists in South Africa and Australia that are building crazy big radio telescopes that use Ceph to store all their data.
“There's a lot of Ceph adoption and a pretty robust community of companies in Europe. I think there's an absolute tonne of Ceph deployed in China. We hear stories about people building these ridiculously big clusters using Ceph, but there's a language barrier and so the communities don't talk as much. But there's no doubt that there are many, many exabytes of storage in China that use Ceph.
“We hear all sorts of stories from all around the world about people who are running Ceph. A couple of years ago I ended up helping out an ISP in Baghdad that was running into trouble with their Ceph cluster. In a lot of parts of the world where cost is a major concern to IT service companies, Ceph - not to mention other open source solutions - are what keeps the lights on.”
Read the full story HERE.
“Sometimes automation is seen as a panacea. We see it as one of a number of solutions, not a silver bullet.”