In this episode Asim is joined by guests Chris Adams, Sara Bergman and Danielle Erickson and they discuss the impact that Amazon’s Customer Carbon Footprint Tool is having on the green software landscape. How do services like AWS affect climate change and what are the effects on the environment of these huge data centres? We also learn about how you can use heat from greenhouses to grow tomatoes!

Transcript below:
[background music]

Danielle Erickson: We're looking at the AWS tool, the Google tool, and the Microsoft tool and understanding the broader strategy, so when you combine those two things, I think that's what we have to do right now to strategize in the best way to reduce our emissions.

Asim Hussain: Hello, and welcome to Environment Variables brought to you by the Green Software Foundation. In each episode, we discuss the latest news and events surrounding Green Software. On our show, you can expect candid conversations with top experts in their field who have a passion for how to reduce the greenhouse gas emissions of software. I'm your host, Asim Hussain. Welcome to Environment Variables, our new podcast. My name is Asim Hussain. I'm the executive director of the Green Software Foundation.

Chris Adams: Hi, there. My name is Chris Adams. I am one of the directors of the Green Web Foundation.

Asim: The Green Web Foundation, the Green Software Foundation, there's a story behind that.

Chris: There is indeed.

Danielle: Hi. I'm Danielle Erickson. I am the product manager of the Cleantech service line at Thoughtworks and the product manager of an open-source tool we created called Cloud Carbon Footprint.

Sara Bergman: Hi. My name is Sara Bergman. I am a software engineer at Microsoft and I'm also the chair of the writer's project in the Green Software Foundation.

Asim: All right. I think we're here today to at least start off talking about Amazon's Customer Carbon footprint tool that they announced recently. This is something I saw they announced in re:Invent which if I remember right, was it November or December in last year, in 2021? They announced it in November or December and it's a method of measuring the carbon emissions of customers' workloads on Amazon.

I don't know how all you all feel about it but I was really impressed with just the speed with which they made their second announcement, which was just, when was it? Weeks ago now or maybe February, they then announced it in preview. It was incredible breakneck speed. I was expecting from their re:Invent to wait a year for them to publish something. For me, that was a really, I felt like I had a whiplash. I was like, "Whoa." Has anybody else had a chance to look at their announcement?

Chris: I have. I think it's really cool. I'm really, really glad that something like this exists because if you consider yourself a responsible technologist, I figure the electricity has to come from somewhere, and being able to have this information allows you to optimize for carbon and given the information that is shared with us increasingly in the news, it's worthwhile actually referring to this. For example, the WPC, so the IPCC report explicitly mentions digital and the role we have to play now in that. Having the tools to instrument that is really, really handy. We actually used to work on something like this. A couple of years ago, we built a tool called Amazon Green Cost Explorer, which basically used some of the billing API to work out which regions were green and which ones were not green, so you can act on this. To actually have something much more fine and green is really, really cool, but it's not the only project going in this field, I suppose. It might be worth talking about that, yes, there are other ways that you can understand the environmental impact if you compute than just using this, for example, even though it's very, very useful and very, very welcome.

Sara: Yes. I think that's a really good point. I think, for example, if you think about performance, some people are very interested in getting into the nitty-gritty. I want to read the logs, I want to really get down to it but others are just I want to see my latency. I think the same goes for carbon, right? For some, this will be revolutionary and this will be a great way to get any kind of insight but there are others who are ahead of the pack where maybe the granularity isn't enough or isn't timely enough or whatever but this isn't a really good first step in my opinion.

Danielle: This is something that customers have been asking for for a while, so to see Amazon respond in this way and give some visibility that I believe is generally available, this tool. Anyone who uses AWS can see their carbon emissions over time is really incredible. It's a great step in the right direction and as Chris already said, there is a lot more regulatory pressure to be reporting on this. Everyone's going to need this and I think if Amazon can make this first step, we can hope that in the future, they'll continue to respond to this demand and this need the customers are having, so really excited about this.

Asim: This is my mistake as well. It's generally available, is it now, Danielle? I just assumed it was a preview.

Danielle: That was my understanding, but Sara, you may know better?

Sara: No, I heard the same. It sounded like it was for everyone who uses their services.

Asim: Yes, it's done.

Danielle: Yes.

Asim: Wow. That is very impressive to go. I mean, it means they were working on this for a long time, I think. You don't come up with a tool like this in two, three months. This has been something they've been working on for a while under the covers, I imagine.

Chris: I think you're right, Asim. I mean, if you've been following this, Amazon have been hiring for sustainability specialists for the last two or three years. Also, if you look at the VP of cloud at the moment, I think, or one of the VP of Sustainability, Adrian Cockcroft, he's been speaking about this for a very long time. If you followed him on Twitter, he's actually had a lot to say about this even in the pre-Amazon days, actually. It's really, really cool to actually see some leadership here on this. I'm quite impressed with this as well because between 2019, there was actually a talk by AWS specifically at a conference called, Map Camp where they were explicitly calling this stuff out.

They said, "Look, if you want to do this, you're going to need to tell us as customers because we're not seeing the customer demand for this." There's a slide of Mr. Cockcroft standing with a big thing behind saying "The thing you can do is move to the green region." Now the thing they've created now is something which provides a bit more visibility to that, so rather than just having that as your option, you've got ways to optimize the actual compute in place rather than having to take on what in many ways could be a risky or scary migration that you might have to weigh up against other things like feature development or the other things that product manager or a CEO might be asking for.

Asim: For me, it felt like the cherry on the top because Microsoft announced their, I think it's now called the Emissions Impact Dashboard a year ago. Then I think it was six months ago that Google announced their dashboard as well and so with Amazon coming out on it as well now, that's all three of the major clouds, all [unintelligible 00:06:39] use that major clouds now have a capability of customers, basically being able to answer questions along the lines of, 'Well, how much carbon emissions is all of my emissions?" This is where I think Thoughtworks would be leading the pack as well because I've forgotten what you call it. I keep on getting confused, Cloud Carbon Footprint. Is it?

Danielle: Yes, that's it. Yes. [laughs]

Asin: Okay. Yes. Cloud Carbon Footprint tool because you've actually been developing essentially an open-source version of this for a while. You must have some really deep insights into how do you actually go away and calculate some of these numbers?

Danielle: It's been really cool seeing the three different tools come out and right now, our team is going through an internal process of reviewing each of the different tools and understanding the variety of features that each of them has is what they have to offer. As much as we can understand how they're getting the data and the methodologies they use to calculate carbon emissions, we're trying to do so because each of them are going to give you really the best numbers you can get for each of those individual cloud providers, but one thing that they're unable to do at the moment is compare between each other.

For many organizations, the majority are multi-cloud users and if you're trying to look holistically at your sustainability strategy and your cloud emissions, you likely want to see them in one place and also using a similar methodology. If you're looking to compare, if you're looking to really optimize, take action, you'd want to compare them apples to apples, not oranges to apples. For our perspective, using all of these tools together is really the best strategy. Have a lot of tools in your pocket to understand what's going on and then begin to understand the areas you can start to make changes.

[music]

Asim: I think we've talked enough about all the glowing praise for all these three tools. We now dig into the issues with them. I think one of them, like you just touched on there is exactly how is Amazon calculating its numbers? How is Microsoft calculating its numbers? How is Google calculating its numbers? There's a lot of opaqueness because they're not revealing that. They're just revealing, 'Here's your total number."

Chris: This is one thing that I could share some light on, I suppose. There is some good news in that increasingly organizations are now talking about essentially, how they share which parts they do measure, which parts they do not measure inside this. We have established ways to track some of this stuff. There are things from the GHG protocol, which is an organization that pretty much sets some standards for this and they talk about things in terms of Scope 1, which is burning for carbon emissions on-site, Scope 2, which is electricity, and then Scope 3, which is stuff in your entire supply chain.

A lot of the time when you might look at some of this, you might have people talking about just Scope 1 and Scope 2, for example, without necessarily talking about the Scope 3 part. If you look at say, I know this is one thing that both Google and Amazon don't include in their numbers is basically the environmental impact from creating the servers in the first place. This is one thing that's probably worth talking about because well, they have to come from somewhere, and it's obviously an energy-intensive process to turn sand into silicon chips.

This is one thing that I've been quite impressed with because there actually are a few open issues on cloud carbon footprint to start piecing together some of these numbers because this is actually very much considered the next step, now that stuff is being done on the energy front. There's a really good blog post in the show notes by David Mitten, who's been writing about this. I'd really recommend his blog because he provides a really, really useful set of incisive analyses in this field.

Sara: Yes, I think that's an excellent point. Depending on the type of application that you have, the hardware emissions just from creating a server or whatever, the network devices, whatever you use can actually outweigh the pure energy cost of it. It depends, of course, on multiple factors, but it definitely can be the case. When we talk about engineering and engineering enablement, there are some pretty easy things that you can do to decrease the amount of hardware that you use, but if you're not getting measured on it, how will you be incentivized to do those actions?

If it's pure cost, well, we are very much relying on cloud providers being kind enough to give us a cost which is mapping to carbon, but that isn't necessarily true always. Right?

Asim: Yes. I think just essentially from my understanding, Microsoft's emissions dashboard gives you Scope 1, 2, and 3, so it tells you the carbon emissions of your workload, your energy consumption, just to break it down a simpler format, your energy consumption and your hardware. Google currently just gives you your energy consumption of your workload. I have actually assumed the Amazon one was all three but is it just energy again? It's just energy again, so Amazon is just energy?

Danielle: Yes, one and two.

Chris: I've read the 451 report. In the announcement, there is a report by 451, and they explain what's in the model and outside the model. They basically said, "We're not looking at embedded energy, and the actual machines themselves, and we're not looking at Scope 3 at present. We're not necessarily looking at Scope 1 because it's not quite so tight." This is primarily about the energy part and this is also why the numbers, as far as I'm aware, there is a lag, because they're looking to get the most accurate numbers, just like how Google do where they basically say, "We will move as fast as we can, but we are working with very, very large providers who might not bill on the same monthly basis. We wait until we have the information from energy providers, so we can give you an up-to-date number." This is what is actually shared to my understanding.

I've got to stress, I do not work at Amazon, so there may be much more detail that may be there that I'm not so aware of.

Danielle: I'm not exactly sure the full reason for that lag but my understanding is it's about three months, which if you're getting very accurate information, can be helpful to look back and understand over the course of the year. I do see a challenge to the consumer actually trying to make changes and use this data. How can I act on data today that's three months old? It becomes a little bit difficult to build into your workflow, to make decisions day to day based on three-month-old data. That's something to consider, I think, with this tool, and maybe something they can improve in the future.

Sara: Yes. What they're stating in their announcement is that it is the underlying billing cycle for the electrics utilities and I believe that Google is doing the same, but they are also quite late in showing. It really limits what you can use it for. It's still great for some type of comparisons. If I have two applications that are similar, which one do I continue with? Those sort of things, it's very good for, or comparing over time, but doesn't really tell you what I should do tomorrow. I think as more and more software companies move away from the waterfall and move into more and more aggressive agile- three months, no, is anyone going to be really happy with that? I don't know, maybe.

Chris: Maybe there's one thing that you can talk about here in terms of, there may be different uses for this data, for example. I know that when I've spoken to people who are looking to use things like cloud carbon footprint, they've told me that there's two main use cases that you tend to have. There might be engineers like yourself or me when we want something like an SLO for carbon, I want to be a green SRE and there's a really nice post by a guy called Benoit Petit, who is one of the lead contributors to a French project called [unintelligible 00:14:34] which basically provides per-process level energy usage information that could provide these numbers.

He talks about this stuff as an SRE saying, "Well, these are the numbers I need to basically optimize for, and I should have dashboards like this." There's some really nice work by the folks at Mapbox, who've been speaking about this for a while. They were some of the early contributors to the early green cross-explorer stuff for this where they were talking about, "Well, if we review our bills on a weekly basis, and we use that to shape our usage, it'd be awesome if we could do this for carbon because we're already good at optimizing for some kind of metrics, so it would be really nice to have something that."

Increasingly, we do actually see numbers like that now. There are schemes which do make this stuff possible. Just last week, for example, there was a new standard which has been proposed called granular certificates by a number of organizations. This gives you hourly settlement for this stuff, which is really, really, really impressive. This isn't that well known yet, but this is the kind of stuff that what the future looks like in my view. I look forward to the time where this is actually a thing that you can optimize for as an engineer, and you can see on a dashboard, for example, for your team.

Asim: I think really, the issue here is that we want to celebrate this work on the work that we're doing, but it's not quite there as a dashboard, that from an engineering perspective, teams can use to actually give them information to make decisions. That's basically the challenge that we've all got. As we're sitting there and we've got options between one, two, and three different architectural types or different choices. This doesn't quite give you that level of granular-- Regardless of even the methodological differences between the different platforms, even the granularity won't give you that.

I can't speak for how Amazon does it. I do have some experience for how the Microsoft dashboard works, and it is very averaged out data. Multiple servers will always report the same energy consumption regardless of what you do because that's just how it's been calculated. That works great the thing you're talking about what is it used for? They're designed for reporting purposes. They're designed so organizations can calculate and report their carbon emissions to CDP or perhaps have an understanding regarding what are the offsets or neutralization strategies we need to employ. That's just what they're designed for. It's not built for engineers with the caveat of-- I think Google is on an interesting track.

Sara: Yes. If you think like a person in an individual team with a small portfolio, then I completely agree, but maybe if your step up, so if you're someone who is responsible for a larger portfolio of services, then suddenly this means you're able to compare them. Sure the data is older, but I can then start to evaluate, "Okay, how much value is this service provided me compared to service B, and how much is their carbon footprint."

If one is vastly higher, but providing me less business value, then that's a decision on a leadership or a planning level that you can take that these dashboards enable that you would not have been able to reach without this. It really depends on what kind of decisions you're trying to make based on this dashboard.

Danielle: Yes, this is something that we thought about a lot when building the open-source Cloud Carbon Footprint tool. Our perspective has been trying to reach that engineer level, that day-to-day decision-making level with as much granularity as we can build in and as much real-time as we can try to make the tool, taking billing data immediately and turning it into carbon emission estimations. Not to repeat myself, but I think the benefit of having multiple options here is that you can combine them for these different uses that you have.

Your engineers can look at both tools, combine the data that they're seeing from the Cloud Carbon Footprint on a day-to-day basis, and then talk to their infrastructure leads who are looking at the AWS tool, the Google tool, and the Microsoft tool, and understanding the broader strategy. When you combine those two things, I think that's what we have to do right now to strategize in the best way to reduce our emissions.

Asim: Especially the fact that because Cloud Carbon Footprint is open source, not only is your methodology public, but your data and the underlying data assumptions are very low, granular level are public. I can see what is the energy-- If I'm using this particular server, this particular load that data is public. We're actually using that in the foundation in the software carbon intensity standard, where you're leveraging that data because it helps engineers calculate the carbon emissions of processes or estimate the carbon emissions of processes, so they can then make those kinds of decisions.

It's the openness of the data is, I think, also missing with these tools. I've also heard it's extremely difficult for Amazon and Google and Microsoft to make this data public. It's not only they're revealing competitive information, there might also be legal constraints. If you reveal some of this information, the SEC might come after you because you're revealing proprietary information. There's actually lots of complications around that, from what I've heard. I wonder if others have thoughts on that, on the openness of data.

Chris: I can actually weigh in a bit on this, which might be of use because you have a similar thing happening in the energy sector just the one layer below right now. One thing that we've seen pushback from people who run the energy grids in various places, they've typically said, "We are not able to share information about how congested various parts of the big transmission wires that move power around because we see that as a security risk." but this is actually a thing that we've heard in lots and lots of places. In many cases, a lot of the time you could see there's a trend towards opened for a bunch of this stuff.

I feel like a lot of the time, if you're not designed or if you aren't used to sharing things open by default then you can come up with a lot of- it's understandable that you might not want to share a bunch of this stuff. There will be cases where you might not want to share this for very valid reasons. For example, there are probably valid reasons for not listing where geographically every single data center might actually be. Even if this may be information that as a customer you might want know if you want to understand what climate risk is associated with all the machines running in a particular place.

Especially when we refer to examples like say- a most recent risk example might be the big Facebook data center, the big data center from Meta and Zeewolde in the Netherlands. That's eight meters below sea level, that gigantic data center. That's the thing you might want to know about in a world of rising sea levels. That's some of the stuff which is useful to know about, but going back to the original point. not everyone's ready to share information on a very, very open basis just yet, but I suspect that over time this will have to come up because well, environmental factors will increasingly push this and necessitate this kind of disclosure.

This is actually one of the things that's been driving a lot of this stuff right now. It's because investors are basically saying, "I need to understand the disclosure in my supply chain." or "I'm invested in you as a company. I need you to share this information so I can end up with a net-zero portfolio. If you don't have these numbers, it's going to be very, very hard for you to share that." In many cases, organizations will basically say, 'Well, I'm not going to invest in you. I'm going to invest in someone else. At least I know whether risk is there." We're not open yet, but the more often we do get, the easier it is to make data-informed decisions as we move forward into this changing climate world.

Sara: Now, I see the same security issues for hardware as well. Do you want to state exactly what type kind of servers are on your server [unintelligible 00:22:25] floors? Maybe not because there has been hardware security incidents in the past, I'm sure we'll see them in the future. Then you might not want to say exactly what you have, but there can also be an argument for finding what is a valid enough proxy that you don't state explicitly that this is this type of server, but exactly this hardware. I built it like this. I specify carbon cost or some other tangible number that gives you the information that you need without being a security risk.

That is, of course, a lot of work especially if we think across all cloud providers even if your company is your own cloud provider while being on Preem, you would want to be able to compare across the stack. The lining on that without being open is difficult. That we're going to guess what our competitors use. I don't think that's a good approach. It's quite exciting from an engineering perspective, just the complexity of some of those things.

Asim: That's a really good point that you mentioned about what data can you reveal? Because this is what we're talking about with the software carbon intensity specification in the foundation is what we want. We're talking at one level about give us all the data, but really, why do we want this data? We're actually trying to calculate our carbon emissions. Well, what really would be quite useful is just the carbon intensity. It's like this server, I don't necessarily care what the components are. I want to know how much carbon per CPU, per minute of this [unintelligible 00:23:55].

I want to have that data, and if I have that kind of data, that's actually probably all that I need from an engineering perspective. That's probably all that I need in order to make decisions. It'll be a wonderful world in the future where everybody is essentially giving you this data is what is the carbon intensity of my service? What is the carbon intensity of this streaming service we're using right now? What's the carbon per minute? That's all I really need.

Chris: It might be worth looking at some work that's happening in the web world that I've seen. There are tools like website carbon and increasingly there are tools that plug into analytics like Google Analytics to give you an idea of what the environmental footprint of some digital services over time might actually be. One thing that I've seen in the web world right now is this real push for having carbon budgets for websites. One company, Wholegrain Digital, they basically say, 'No website that we build will cost more than two grams of CO2 emissions per page load. That sounds really silly on a per-page load basis. Some websites get quite a lot of page loads, so over time this stuff adds up.

If we just zoom out for a second and think in the outside world, there is a huge amount of science saying, "Yes, we have a budget we need to stay inside." If you look at the energy sector, they themselves have a carbon budget that they have agreed to stay inside which is why you have massive compliance markets. It makes sense. Probably we would also need to have something like that as well if we want to stay inside, like I guess the dictates of science. We don't get to change the physics of climate change but we can at least change some of the economics around climate change.

We can at least do something around this so we can optimize for carbon as developers, so when we're building services, they tread more lightly or as lightly as possible given the various other requirements would be nice to meet as professionals, I suppose.

Asim: It's interesting that you mentioned the carbon budget as two grams per page load, because that's an intensity, not a total. I think that's the thing that I talk about a lot that total budgets are really challenging in our world because there's just- how can you set a one-ton budget for our website? You have no idea how many users are going to land on it, but an intensity is, "Oh, [unintelligible 00:26:10]. '"

Chris: I'm not so sure about that.

Asim: Oh, interesting. Let's go.

Chris: Here's the thing, Let's say you're going to go with this. You have a $2,000 budget or something like that because you know that you're probably going to get this many page views over a given time. This is the thing that you're seeing in procurement and contracting these days. They're basically saying, "Well, we have been given legally binding targets that we need to reduce our emissions by 5% a year, a year between now and 2030. That's it. We don't get to do not do this now it's in the law. If they have that, then they're going to have to say, 'Well, we're going to spend €100 million, £100 million on this contract for the next two or three years. We basically have an implicit budget that we need to stay inside."

You do have something like that now. It may be the case that okay, having just one number over three years isn't very useful. You might want to have some smaller timeframes or something like that. This is why it might be useful to have a rate for this because you can say, "Well, given that I have this, I now have something I can act upon. I can either change the size of a page for example or I can change the intensity of the electricity so that's going to allow me to stay inside it. It gives me more options." I think it's useful to have the total number because this is essentially what's driving things from a science and regulation point of view. As a developer you might not be able to use it on a daily basis.

If you have CI for example, you're going to want to have a unit because that's what you're used to using for your score from say, Google's web vitals. A web vital score is going to be a rate that you can refer to or something you can look at. It's not going to say over six months. It's a kind of volume basically. I think you need both, Asim, not just one, but it's very, very useful to have the ratio, absolutely.

Sara: I think tools like the tool we've been talking about today, Amazon's new tool, that can give you that from an OKR perspective because you can see, "Okay, what's my cost? What's my page views? I do the simple division and I do get these numbers, but once again, it's for reporting purposes. If you've never reported on it, this is better than not reporting on it for sure. Totals are also interesting because you can go to a rate assuming you have the other end of the fraction, but sometimes you want to go the other way. That can be a lot trickier without the really granular data.

Chris: Sarah said something really interesting here about going both ways and about if you've got a total number, you can go down from there. I've mentioned Wholegrain Digital before because I'm a really big fan of what they do and EcoPing is another group that do this stuff as well as Mightybytes who built a tool called Ecograder almost 10 years ago where they were tracking this kind of stuff. The model that is used, they call it the sustainable web design model that is basically based on a global figure for all the initial used by all the entire internet tech sector divided by all the data transfer that is facilitated by this.

This is a bit of a course figure, but at least gives you something that you can act upon and work with. This is actually one thing that I think it's going to be live next week, as being able to use some of the tools that you do use if you build websites and have things like that. It's useful to have those kind of stuff. In many cases, you need to understand what the model is actually representing to see what's going into that, for example. In the example of CO2-GS, for example, this is using network transfer as a proxy to talk about things like, say, usage at a device level or usage at a data center level.

Once again, without having access to the open models, it's very difficult to know where your interventions are going to make a meaningful difference. This is why I'm actually quite happy that things like Cloud Carbon Footprint are open enough and are accepting [unintelligible 00:30:07] requests. You can basically say, "Well, this is what I think is going on. This is what I'm trying to do in good faith to reduce the emissions of whatever service I'm building."

Sara: Yes. I think we should also maybe mention that the granularity only matters if you have an application or service that changes rapidly. Not every software does that right, that we have stable software that's in a maintenance mode or for whatever reason, isn't that interesting to change frequently. Then this is honestly good enough. You don't need hourly data for a service that you're going to update twice a year. That's not needed. I think it's interesting to compare to, for example, the transportation sector.

I was taking a train recently in Norway, and on the app, they showed me by taking this train, you've saved this much carbon. I got annoyed. What is this magic number? Then I clicked on it and they actually showed the entire methodology. I'm like, compared to flight, if ou are one person, da, da, da, this is how much. If you were to take one four-person car, you were one of those four people, this is how much you would save. I was pleasantly surprised. The reason they can do that is that is because the cost of fuel and the cost of building a car doesn't change that frequently. They can calculate this once and then use it for a really long time. For software that's tricky. You have rapidly changing software, so this is something to keep in mind.

Danielle: The thought that you brought up, Sarah, about stable applications made me think of a trend work seeing with usually more mature organizations to try to start understanding, if I have stable applications and I also have typical architecture practices that I continue to use, can I start to understand particular architecture choices that impact different carbon and energy use? How can I understand and learn from those architecture choices over time, and then maybe even automate that process? I learned that this particular architecture choice is less carbon-intensive. Can I just make a dashboard or facilitate provisioning certain services in that way?

Asim: Danielle, I think you're right. This is one thing that we haven't really got around to developing a language for yet about how you optimize for carbon at various places. in the show notes, I've shared a link to a thing called the Green Cloud Triangle, but we've spoken about this. There's a kind of iron triangle of compute cycles, response time, and cost that you might want to be doing trade-offs of. For example, there may be cases where you want to optimize for response time and cost. This is stuff where you might say, if you want something to be cheap spun quickly, you might go for say, static pre-build stuff.

For example, you're not doing too much dynamic stuff. This is stuff we know already a lot of the time. It may be that there are some cases where you don't need to have things happen immediately, right? You might be more interested in keeping the costs low, but making sure you've got lots of compute cycles. We might use this in terms of having queues or tools or things like that.

Then finally, this is like the default that a lot of us end up using when we're not thinking about this, which is basically optimizing for compute cycles and response time, not really thinking too much about the cost part or not really knowing that the cost can change in this way. This is like speaking to the fact in many cases and what I think we're going to see more of every time is that the cost of electricity changes depending on the time of day. This is not really exposed to us right now, but it's something that is definitely visible, that does definitely happen especially if you look at the markets such that sometimes the cost of electricity can go negative, so you can be paid to use Compute, for example.

I feel like there needs to be a set of tools or a way to describe this stuff so you can take advantage of these changes that have been happening one layer down in the stack, so that you can basically architect for better, more responsive things, but also in a way that's actually very. very planet-friendly as well as wallet-friendly. I think there's a couple of good posts on the Green Software Foundation blog specifically about this. I might have written one of them, but the other ones have also been written by other contributors.

[laughter]

Sara: Well, I can speak about electricity because I think that's interesting. Many of these cloud providers, they say that it takes a long time for electricity utilities to get to them. During this winter in Norway, we only have hydro here, so if it doesn't rain, electricity becomes expensive or if it's really, really cold, all our water is frozen, it becomes expensive. There were these newsletters reporting almost hourly on like, 'This is how much it costs to charge a smartphone right now. This is how much it costs to use the oven right now." Obviously, we know what the electricity costs right now. I wonder how hard it would be to propagate that.

It doesn't always correlate to carbon though. My carbon cost was the same throughout the day because it was hydro, but their cost changed.

Asim: Yes. If cost- [unintelligible 00:35:05] cost is a proxy for carbon, really at the end of the day, I think here.

Chris: All right, you're talking about cost as a proxy for carbon, and that's because a lot of the time when you have more electricity than you need on the grid, it's lunchtime and it's sunny as hell, or it's windy as hell, you've got more than you need. The problem with the grid is that the grid has to basically be balanced the entire time. Otherwise, basically, very, very bad things happen and very expensive hardware gets damaged that falls off-grid. You can end up with incentives to basically-- If you operate a grid, it's cheaper. It's easier for you to basically just set the cost of electricity to be lower than it is to ask someone who runs a big nuclear power station to please turn down the output to make sure stuff is balanced, for example.

This is the thing that you are often doing. This is why the cost will change over time, depending on how much demand there is compared to how much supply there is. Most of the time we're shielded from this, but it's actually quite fascinating. It's stuff that you can absolutely take advantage of because companies do this. Google make a really big thing about shifting loads to when energy is green, but the reason that they do that is it really saves a bunch of as well as just carbon, basically.

Asim: Yes. That's the secret to a lot of our spaces that there is cost savings, a lot of this stuff as well.

Sara: I don't necessarily think that's a bad thing. Capitalism runs large parts of this world. If we can get those forces to work with us, promote us to a greener future, we shouldn't necessarily be against it. I'm saying capitalism is always awesome, but you take the wins where you can find them.

Chris: This speaks to incentive design, basically and who's making good use of this. There's a really nice example of organizations. There's one organization in America called Lancium, and there's another one in Germany called XMesh. They take advantage of this. They basically take data centers which is basically a shipping container full of machines, which will otherwise be thrown out from hyper-scalers like Facebook. They put them on renewable energy parks. What they end up doing is they end up providing stuff for either-- okay, I'm not a huge fan of the cryptocurrency stuff, but you can use the same thing for machine learning models as well.

Anything, which is a plausible load that is quite compute-intensive is a really good fit for this use of an oversupply of renewable energy in many cases. This is what Lancium and XMesh both do now. By being able to be plugged straight into places where they have energy, that they otherwise would not be able to use, they basically end up being able to provide Compute for a much, much lower cost. You can get your machine learning models done at a fraction of the price from some of other larger providers by going with this because they're taking advantage of the economics and how they've changed over the last 10 years that in many cases say some other there haven't taken advantage of yet.

Asim: Why is that more economical? At the end of the day, those servers are servers that they got essentially maybe for free or very, very, very reduced cost. They must be four or five years old.

Chris: There are two reasons. Moore's law has slowed down over the last, say five years, for example, before you could just rely on [unintelligible 00:38:20] to do this work. As a result, servers which are maybe two or three years old, aren't actually that much slower than they were previously. If you've got something like a plausible load, because you are not trying to run it 24/7 all the time, if you have different requirements for keeping this stuff cool, for example. Unlike, if we're going to talk about keeping cool, for example, there are some really nice examples in the Netherlands where they basically have shipping containers full of servers.

Once again, these are servers which are end of life. They plug them into greenhouses with the idea being that the waste heat, rather than basically vented into the sky, or you spend loads and loads of money trying to get rid of it because you see it as a waste product, they use it to pump into greenhouses, so they end up with really nice juicy tomatoes. This is a really, really cool use of heat because the greenhouse folks, they were like, 'Well, we can either burn fossil fuels for heat or we could just use that heat from over there." This is an example of taking advantage of-- If you understand the underlying energy systems, then there are all these fascinating, new, pretty cool use cases. I don't know about you, but the idea of, I don't know, a greenhouse connected to a data center and juicy tomatoes, that sounds cool. I like that idea.

Asim: I do know, actually that heating greenhouses is one of the biggest costs for greenhouses. That's wonderful. I love that.

Chris: There are loads of examples here. When you look at the next challenges we are facing between now and the next say, five to 10 years, one of the big ones is heating things up. What we have right now is we have a massive data street full of data centers thinking, "I've got all this heat, how do I get rid of it?" It feels like maybe the people saying, "Well, if only we could find a way to get heat." and then we'll tell the people saying, "Oh if only I could go find it, we'll get a way of getting rid of this heat." If they could talk to each other, then maybe you could end up with a slightly more efficient system.

Now, this isn't going to happen all the time because if you put a gigantic hyperscale data center miles away from everyone else, it's going to be harder to integrate that into in an urban environment. Then maybe that speaks to the fact that our idea of what a data center needs to look like could change over time to end up with a different topology for the internet because the internet did use to be quite distributed. What we've seen right now over the last 10 years is that the energy sector has ended up looking a lot more like the internet and the internet is now looking a lot more like the energy sector was 10 years ago.

I feel like maybe there's scope for us to find some happy medium rather than just zipping past each other in mad decentralization or centralization mania that we have at present.

Danielle: I think, Chris, with all that you're saying there's so much opportunity. My question is, where does the responsibility lie to provide that information to consumers, and who is responsible to make these choices of shifting workloads, taking advantage of the energy at different times of day, that type of thing? How much can the cloud providers do and how much can the consumers do and what is that balance? How do we get there? I think it's going to be a really interesting problem that, hopefully, we get to solve in the next few years.

Sara: I'd love to see a carbon throttling thing that you can add to your services, whatever cloud provider you have. It's like, Yes, you can carbon social this application. That's fine for me."

Chris: There is loads of cool stuff happening in this field right now. Brunch Magazine does examples of this. If you go to brunch.climateaction.tech, it throttles based on the carbon intensity of the grid right now because this is exposed to it. There are also tools [unintelligible 00:41:49], for example, that let you do this stuff. This stuff exists and there are examples of it being built. I think it's a really exciting fun place to be looking at this but there's a whole policy piece that would map to what we're doing here.

[background music]

Asim: This has been a wonderful conversation. I love all the places we've been to. Maybe let's just end with just one quick thought or idea from each of us and think about the future and something from our conversation. I might start because if I don't, I'll forget. There's something you touched on, Chris, earlier on, I thought was fascinating. You talked about what of Meta's datacentres was going to 8 meters below sea level.

One of the things for one of our future podcast episodes I would love to explore is the SEC has just had a proposed I think ruling, I don't know if that's where you got the data from. The SEC has proposed ruling now that organizations have to disclose their climate risks. I'd love to have a conversation about what are the climate risks related to software and green software and sustainability and technology? That's a great example that you gave, and I just thought that's something I'd love to explore in the future.

Sara: Final thoughts, placing my tomato plants next to my laptop, number one. Number two, it will be interesting to talk in the future about how the pure economical aspects of where to place a data center will impact the grid. If you're only placing data centers where the grid is green, will that power a green shift in the energy markets?

Danielle: I'm having trouble wrapping up all these thoughts. There were so many different avenues. I think something that stuck with me that I'll continue to think about is the idea of carbon intensity and viewing that in conjunction with totals, using these variety of numbers to come up with a strategy. I thought that was really interesting.

Chris: I guess that's me left now, actually. Asim, I'll keep it short. I think this points to us having a carbon-aware internet. I think that's a really cool vision, personally. I'll leave it with you, philosophwith you, Mr. Hussein.

Asim: Thanks for listening to Environment Variables. All the resources for this podcast, including links to our guests and more about Amazon's customer carbon footprint tool, as well as the Green Software Foundation, and everything else we read discussed today is going to be available in the show description below. We hope you enjoyed the show. See you on the next one.

[background music]

Asim: Hey, everyone, thanks for listening. Just a reminder to follow on Apple podcasts, Spotify, Google podcasts, or wherever you get your podcasts. Please do leave a rating and review if you like what we're doing. It helps other people discover the show and, of course, we want more listeners. To find out more about the Green Software Foundation please visit greensoftware.foundation. Thanks again. See you in the next episode.

[music]