Episode Transcript
[00:00:00] Speaker A: Kafka is a proven technology and can be one version of the truth for data, and it allows you to really act on events when they happen or even before they happen.
[00:00:24] Speaker B: Welcome to the EU Energy Projects podcast, a podcast series from Enlighten France focusing on the clean energy transition for the European Union and the EU Commission funded energy projects that will help us achieve it. My name is Areti Daradimu. I am the editor of the EU Energy Projects podcast and your host.
[00:00:48] Speaker C: In our previous episode, together with Laurent Schmidt, we discussed Project Eddy, the first data space to bring real time consumer data into markets, as well as next gen data spaces. In this episode, Loran and I are joined by Kafka's CCO yellow builders. Together, we are going to focus on open source data exchange and especially the Kafka for power data exchange platform, but also the new white paper future control room architectures for grid and market integrated operation. It was created by actual and digital for grids, and you can find the link to it in the article that accompanies the podcast episode. Finally, Lauren Yele and I are going to discuss how projects like Eddy and Onenet complement a data exchange platform like Kafka for power.
[00:01:45] Speaker D: So my first question is, what is a control room and why is it so important?
[00:01:50] Speaker E: So I see control room like air traffic tower control, which is really managing flows across the entire system, congestion and basically linking with any large assets which have impact on their flows.
The energy transition is evolving this environment towards real time, hence the need to expand the reach of these control room environments towards lower assets.
[00:02:26] Speaker D: Okay, so what is Kafka? How is it connected to control rooms and other energy projects like one ATI?
[00:02:33] Speaker A: So within integration, the use of real time data becomes much more important. And that's mainly because of the fact that think of smart meters. If a lot of people at the same time perform a particular action on a smart meter, so they turn up the heat, for example, it's very important for energy companies to know exactly where that will be. And do they have enough energy that they can use for that? Or do they actually have to sell off energy to other energy companies, for example? Well, this only works if you're able to get that information within your organization real time. For that, we use Kafka. Kafka is an Apache technology that's been created quite some time ago. With that, we can stream particular what we call events. But the good thing for that is if you store all of those events on one particular platform where you couple all your applications that you use within an organization, you're going to be able to steer the ship directly real time based on events that happen at that particular time. So if the sun goes out and solar panels do much less, it might be that we need to get energy from different places.
This you will then know exactly at that moment and then you can take action.
[00:03:59] Speaker D: So if understood correctly, the white paper highlights a shift towards consumer centric and smart energy system. So how can data exchange platforms empower consumers and facilitate their participation in the energy market, impacting also however, in that way the control room operations?
[00:04:21] Speaker E: Yeah, I think it's very central to this transformation which we are talking about is the fact that basically to manage balancing or stability of grids, as well as to manage congestion, in the future, a grid operator will have to rely on to flexibility being providing behind the meter as what was just mentioned by Yele, and as near as possible to real time. Because as we all know, it is only close to the actions when we know whether an electrical vehicle is available to provide bidirectional charging, or whether you've got a cloud over your local home which is having an impact onto your solar pv. So it is really a new paradigm of getting these real time information streamed. So that means is not the grid operator calling for this data, but the data is made available based on a local site conditions, so that basically grid operator can make their stability and congestion management decision on the basis of data coming from the consumer environment. So this transformation is really about orchestrating platform across consumer flexibility aggregator and grid operators.
[00:05:52] Speaker D: Speaking of real time transactions, I did read in the white paper again that there is a need for real time transactive interfaces across systems. Now can you elaborate on how this will be integrated within existing control room infrastructure and what challenges that might present?
[00:06:12] Speaker E: It's a very good question on the migration aspect and what we elaborate in our white paper is the way historically control rooms have been designed by historical ScADA vendor whose tendency has been to close themselves to other boundaries than just the grid substations and so on. And so what we call for into these environment is to set up a new data exchange environment based on Kafka, because we think it is the required level of real time interaction to open up these environments and enable grid operator to accelerate some of their developments towards integrating data from the edge behind the meter of their system. So that's what has come clear from the eddy projects in terms of data spaces that we do need a new common european energy data space to be put in place to favor interoperable data exchange across the energy value chain. But this data space must be real time. And so that's the idea behind this partnership, is to open up, is to base it on open source, and is to make it as efficient as what we've experienced with other sets of data coming from the grid substations.
[00:07:46] Speaker D: Yele, I would like to ask you, how will the control rooms transition toward automation and machine learning? How can the control room personnel be effectively trained and prepared to adapt to these changes and work alongside AI powered system? And how can kafka help in that?
[00:08:06] Speaker A: So one of the challenges of integration in general is that it typically requires quite a deep technical level, technical knowledge in order to integrate specific systems.
In this particular case, KAFka allows for a lot easier integration or enablement within an organization because of the fact that there's a platform in between that does all the technical work, let's say underwater.
What we typically do is we train organizations, and not so much just IT teams, but business teams as well, because we have created connectors that are just plug and play where you can have various sources throughout your entire business connect to that technical Kafka platform.
But it's not so much an IT project each and every time. It's just setting up one connection with an existing plug and play connector, and then the data will land where we typically feel that these type of projects get more challenging.
It's in the governance side of things, data lineage, for example, who owns what particular data? Is everyone within an organization allowed to see all the different data sets from all the different applications that you run? Who owns what, who decides who gets access to a certain topic, to a certain event stream? Who's able to see all that data? Or should it be masked, for example? Those are typically the things where we feel that needs more time. But it's not so much on the IT technical aspect of getting an integration.
[00:09:42] Speaker D: I think you touched upon a very important ethical matter, also of who owns the data and what we do with that data. I would like to know your personal opinions on that. I know you're more on the tech side, not the ethical side. However, I'm interested to hear what you think about that.
[00:09:59] Speaker A: I mean, data typically, what I personally believe is that data is owned by a person or by groups. So if we beforehand establish who within a certain organization is allowed to see certain parts of information, then you can kind of like have that distribute data lineage or governance by itself, right? If a group, for example, owns a specific topic, then that group typically is perfectly capable or well equipped to say, well, this other person or this other group might be able to see this information as well.
From an ethical standpoint, I'd say if it's really deep personal information about a specific persons, individuals, then data should be masked, unless that person really needs that information to do his or her work in the energy space. Usually a lot of information is somewhat personal because it has an address, et cetera. But it doesn't go deeper than that. Right? It's not, for example, a bank. It could tell you something about how much energy is consumed or not, but it doesn't necessarily say, what is your bank balance. So I'd say, from an ethical standpoint, energy companies within their own group, within their own organization, should be able to widely share information.
[00:11:23] Speaker D: Laurent, do you agree with that? And do you think that the European Commission and local authorities do enough to protect data? And should they be the ones making the decisions or the companies themselves?
[00:11:37] Speaker E: Okay, so I would maybe slightly adjust what Yele mentioned regarding privacy, in the sense that what we see is a lot of these new flexibility. Business models are actually emerging from the residential consumer markets, where we do know that if we align real time with road profile, you end up into highly private data. So basically, managing consents for these environments are also very important element as a first step and to the point of hire is being able to orchestrate data environments so that you exactly know basically where you manage private data and how you should protect this specific private data as compared to environments where you aggregate these private data at a certain scale, so that you know that this private data has become anonymized by essence, in which case you can make better use of this data. But the data act today is defining, I think, quite clearly how we should approach these aspect of data privacy.
What we would need to do is go probably one step further. And that's one of the purpose of the data space in being much more practical on defining for each type of data being energy, power, capacity, and so on, what is private and what is not private, and what are the conditions under which grid operators are allowed to use this specific set of data. What I do believe is in the current context of grid congestion, and more and more private consumers being constrained by congestion, we hear in some countries not being able to build schools or houses anymore because of congestion. I think there will be a new play with this privacy aspect in the sense that if grid operator consider new flexible grid connection against a certain access or even private data, there is so much appetite to connect to the grid nowadays that there will be new data agreements existing between consumer and grid operator because of the benefits of that private data in basically enabling an improved grid operation, grid optimization, or running the grid at the closer to its limits, maybe.
[00:14:21] Speaker A: To add to that as well. I think there will be specific levels in data privacy, in data security especially.
Well, let's say the two biggest buckets are how do you use and make available data within your own organization. So within a specific TSO or DSO, for example, versus how secure is data stored to the outside world.
And those two are obviously very big differences. I think the latter one you can often cover by making sure that there are the right certifications, the ISO standards, et cetera, are in place, and that there are good agreements and penetration tests, et cetera, performed on the it aspect of things.
[00:15:12] Speaker D: How will this happen? Will it be, let's say, a form like the cookie form that we have for websites?
[00:15:20] Speaker E: Maybe a first part to this answer is that none of the control room today are enabling such kind of privacy by design, integration of consumer data into the control room environment. So I think one of the very fundamental points is to be able to first integrate subsystems, managing different sets of data. And against what Watieri was saying, this integration layer is very important to be able to cluster data environment with different sensitivity of data.
What I think we will see emerging, and we clearly saw it in our last two, three years demonstration project, is new tools to be able to integrate data from consumer, where by essence we will expose a new API to consumer and we will clearly and transparently explain for what purpose this data is being used for. And I think one of the challenge in our industry, as well as broadly in the data industry, is to move away from these cookies and tons of, you know, t and cs, which no one is really interested in, into something which is more simple, but which clearly articulates what is the purpose for which you are asking me my data and what are the measure you take to protect this data. And I'm very sure that given the importance of this grid connection, this grid flow, this grid congestion, in tomorrow's business model, a lot of consumers are going to be willing to further exchange this data against the assurance that it is well controlled, that it is protected, and it is not misused for unnecessary marketing reasons, for other purposes. So that is really what we need to reach in moving forward into these new control environments.
[00:17:28] Speaker A: If I look at consent and the pros and cons of sharing personal data, my belief is, and I have this at home as well, right? I have a Google, my home is full of Google Home screens that supposedly should help me, right? And I've said yes to any consent. Sure, you can listen in on things. I'll give you information. I'll plug other devices into the system as well, and you'll be able to read my data. But in return, I do expect that you keep that safe, that information. Plus you provide me with something value as well. So, for example, before I go out in the morning, I'd love to get a notification that says, hey, jele, you're going to Amsterdam, for example, for a business meeting, but there's a traffic jam you might want to consider to or go at a different time, et cetera. Then it actually provides value. And that's exactly what I would expect from a system as well. If you take that analogy and apply it to the energy sector, I would love to get a notification. If I say yes, you'll get my data. Then also let me know when it would be the right moment to charge my car, for example, to avoid congestion in the network. So there are, I'm fine. Personally, I'm fine to share my data, but I'd like to, in return to get value from it. Plus I want to have trust that my information is stored safely.
[00:18:54] Speaker D: So we discussed a little bit about the security regarding individuals, the end consumer, how we can ensure it, etcetera. Now I would like to flip that and ask, how can data exchange platforms ensure secure and reliable data streams for control room decision making?
[00:19:11] Speaker E: Okay, so I think behind that security aspect, you have the aspect of not being one system, but federated of several systems across the value chain. So it's not only about thinking on your own. Control room security itself, which is something very fundamental. Control rooms, because of the criticality of these environments, are probably by far the most secure environment, but is also thinking about that environment in view of some of these data coming from basically hybrid cloud environments and really being able to combine an end to end security. And I would like to refer this back to why we digital for grids decided to work with a company like actual is because they do have really a very strong expertise into thinking on Kafka systems, which are not a sub kafka system of closed environments, but really of a Kafka system which expands across various pieces of the organization and even eventually to the outside world of the flexibility aggregator and so on. The complexity here is to integrate different security measures from different environments. And that's where basically actual has brought a layer on the top of Kafka, the so called governance layer, which is a very interesting environment to secure that, and it is operational also. One thing is Kafka has a lot of advantage, but it has a disadvantage to be sometimes complex to deploy, while actual has really gone far into automating that kind of deployment, and so enabling a fast adoption by utilities.
[00:21:10] Speaker D: Jele, can you tell me if a system like Kafka can facilitate seamless communication and data exchange between different control rooms also like tsos, dsos and market participants?
[00:21:24] Speaker A: Sure.
Yes, Kafka can facilitate that because it'll be the central data hub, and the sources for that data hub can be anything. Those could be small consumer facing applications, and it can just as easily be a control room and then various control rooms in different countries, for example. So that's definitely possible. And I think one other thing we touched upon earlier is how can you then make sure that that data remains safe and secure?
That's just a really important topic.
There's two ways to look at that from my perspective. One is from an IT technology perspective, which in this particular case means there needs to be certain industry standards, there needs to be certifications, penetration tests done, etcetera. But there are a lot of companies doing that, and it you can quite easily audit, right? Whether it works in a safe manner, whether it follows certain protocols, processes, etcetera. If you look at data breaches, typically the biggest challenges are human errors. That's where it usually goes wrong in the integration space. Therefore, things are different because you set up an IT or a technology system in a way where it adheres to all the standards and the certifications that are relevant in a particular industry.
And after that, you use a platform where people log into, everyone logs into the exact same portal, which has all the security straight off the shelf or out of the box. So therefore, people cannot really break the system, if you will, unless their computer gets stolen or there's a different type of breach in a control room itself, et cetera. But apart from that, data is safe.
[00:23:23] Speaker D: And how can projects like the EU funded eddy and one net can complement perhaps the platform, the system?
[00:23:33] Speaker E: Yeah, so very good question. What we've been looking at through this project is a lot around interoperability and reuse of industry standards, and expanding some of these standards, particularly to connect with distributed energy resources at the edge of the system, which are equipments not coming from the control room environment. So basically, the complexity is to bridge sectors which are not necessarily used to work together. And so I think what is interesting in this project is there are models of data exchange and data standards which are being developed there. And the idea which we have is actual in creating Kafka for power so it's not only about reusing, I would say, the Kafka open source layer, but it's embedding into Kafka for power some industry specific standards and security measures, which helps tsos and DSO to accelerate their deployment. So really what is very important is we are at a time where it and grid integration is potentially becoming the bottleneck of the entire energy transition. And so what we want with this kafka for power is to be an accelerator of that integration, both from a doctor, physical assets, but as well as from an iT point of view, the.
[00:24:57] Speaker A: Central hub for data exchange amongst TSOs and DSOs, and specific control rooms in, let's say, for example, Europe.
That can easily be done with, let's say, Kafka being the heart of that. And it's needed as well, because we have energy trading companies as clients. We have, for example, the entire dutch TSO DSO landscape as a client as well. And I hear quite often that they're selling energy or they have shortage, or there's too much energy at certain moments in time throughout the day, and they're selling that off to, for example, a TSO or DSO in Norway, which is fine. But if all of those tsos and dsos are connected to the exact same system, you can do that in real time based on specific events that happen regionally in certain countries. And if you get that close to what actually happens within countries or neighborhoods, even, that's going to help us really to solve a large part of congestion.
The Netherlands is one of the most challenging markets at the moment, but this will go into other countries as well. They're going to be next unless we solve this challenge. And I believe having a data hub for all these connected companies is going to solve a large part of this challenge.
[00:26:22] Speaker D: I would like to ask a little bit what is happening with cross sectorial integration, because that's also an important aspect for the clean energy transition. Can you tell me how data exchange platforms and the projects that may accompany them and help them out could integrate data from different sectors, for example, power, heats, transport, etcetera, and improve also control room visibility across the entire system?
[00:26:49] Speaker E: The energy system, yeah, it is a very fundamental element.
It goes back to flexibility as a wall is how can you increase the amount of renewable into the power system? And that means basically seeking flexibility in every layer of the power system. So means every sector.
And that basically layer is precisely enabling this flexibility trading down to real time, whether it is for building heating, whether it is for EV bidirectional charging, or even simple charging for electron mobility, whether it's for an industrial site, basically operating its manufacturing in a flexible way. And so we really see these layers being a very good potential for all these environments in general.
What we see is all these environments have their own sectorial standards and environments being defined. And so really these orchestration of standards, you can see the Kafka as, as the vocabulary integration between these various sectors and we can do it in real time. So that is really a powerful tool. And I would like to add as well that the open source element is important here because these sound hall data hubs of course are very critical to a lot of players. And you don't want basically one it company to come and take hold of this entire central layer. So the reason of us believing and trusting Kafka, it is built bottom up from open source. It is scaled, the scale is demonstrated and it prevents basically any big it company to take over the entire domain of transaction across the entire sectors.
[00:28:57] Speaker D: I wanted to ask you, what are the challenges that you faced in two ways. First, the white paper and secondly to apply what white paper says to a control room.
[00:29:07] Speaker E: So these control room environments are exposed to a much larger amount of data flowing as a result of the energy transition and the fact that simply grids needs to interface much more distributed device as compared to before. So the idea of that partnership has matured for the last would say one to two years. And on the fact that the existing technology being used in control home environment were not necessarily open and fast enough to be able to absorb this new avalanche of data through real time environments, and seeing that TSos and DSos had to migrate at a faster pace, this technology to be able to manage their congestion moving forward. So in the future, a grid operator will need to interface consumers edge devices through real time beyond just looking at their own grids and substation, which really requires to open up new interface, new APIs and rethink basically the way these control room architecture have been built historically.
[00:30:25] Speaker D: We started with the challenges. I would like to finish with opportunities.
So I would like you to tell me both. What are the most promising opportunities you see for data exchange platforms to transform control room operations and support the clean energy transition?
[00:30:42] Speaker E: I think the flexibility code, the demand side flexibility code is going really to trigger new data exchange into control room environment. That's really what we see into the drafting of this code. It's going to introduce a lot of new standards and it's going to be a race. It's going to be a race to make it available if possible, in the next, let's say four or five years. So that means before the 2030, so that we can accelerate the scale of renewable integration. And I really think that this Kafka for power and in general, Kafka infrastructure will be an accelerator into that. And that's going to be a way of productizing all necessary tools in the control room, in these future control rooms and accelerate the pace to watch. They can be put online to be ready by 2030.
[00:31:37] Speaker A: What I believe this the collaboration will be able to do is, is to transform the energy sector from reactive to proactive. So now we really know what happens, when it actually happens, and we can then respond quite fast. Right? So it's not that we're ancient, but I do believe if you apply AI and you have a data hub that is able to work across all the silos that might exist today, then you should be able to predict parts of the future. And that predictability is really going to set things apart because that's going to avoid a large part of the congestion, it's going to avoid problems existing in the first place. So I think that predictive capability will be a real game changer.
[00:32:26] Speaker D: Thank you both very much.
[00:32:28] Speaker B: You've been listening to the EU Energy Projects podcast, a podcast brought to you by enlit and friends. You can find us on Spotify, Apple and the enlit World website. Just hit subscribe and you can access our other episodes too. I'm Areti Daradimo. Thank you for joining us.
