AI for all: building technology that leaves no one behind

Caroline Béguin: So let’s start with the big picture. AI is becoming more embedded in our daily lives from hiring to lending, and so do the risks of reinforcing old inequalities. But beyond fairness, there is a surprising link between bias and sustainability. Can you tell us how these two are connected? 

Bettina Vaccaro Carbone: From a sustainability point of view, I think that is something… that’s something that we need to keep in mind and that we often miss, is that sustainability is not just about the environment or being green or maintaining resources. It’s also about how we create an economy that as a whole is gonna be able to continue growing, and that is… It’s not that it is inclusive, but being inclusive is part of the ways in which we support continuous growth and an economy that is gonna be lasting in the long term. If you take the world fifty years ago, and it’s a bit difficult to say it in the current moment, but if you take it fifty years ago, we were excluding almost fifty percent of the population, which is of course is gonna have an impact in the way our e-economy can grow as a whole. And I feel like I say economy every two words. But the idea is that we cannot put a significant part of a population, and of course, this is women in the bias that we’re talking about today, but this is also about other underserved communities. It can be about minorities in general. It is also about people with handicaps and things like that. So every time we exclude a part of the community, we are reducing our capacity to grow by limiting the resources we can actually harness. 

Dana Lunburry: When we’re thinking about gender bias in AI, we can be talking about the ways in which the AI systems are pro-producing unequal outcomes that are based on gender. So often this is unintentionally, but it has real world consequences. Now, when we start to think about the two topics together, so gender bias in AI and how does that actually relate to environmental sustainability, this is actually a very powerful yet overlooked question a lot of times. So gender bias in AI and environmental sustainability, they might seem like completely separate issues, but they’re actually deeply interconnected, even at the root of where they come from. And we can see this through the lens of equity, impact, and systemic design. So let me talk a little bit about the roots of these two issues. The reality is that they share a  certain history when it comes to systemic inequality. So both gender bias in AI and environmental degradation are symptoms of systems designed without inclusive representation. So AI systems, they’re often reflecting the values of the dominant group, often ignoring women in marginalized communities. In fact, the environmental cost of AI, so we can think about water depletion, energy use, all of these things, they’re often falling on low-income or indigenous communities located near data centers or mining operations. These are communities that are frequently underrepresented in AI design and governance. And then when it comes to climate stress, we can think about food security and migration, the insecurity around food rather, this is often hitting women and other marginalized communities hardest. And AI systems that aren’t accounting for this, they can actually amplify the vulnerability of these populations through biased resource allocation or the exclusion from support systems. So some of these connections are just so deeply rooted. I think it’s really important that we think of them in, in that connection when we talk about it and there’s so much we can learn from thinking about that connection as well. 

Shalien Kishore: Thank you, Dana. Thank you, Bettina, for the comprehensive understanding of gender bias in AI. I’d like to highlight first that AI is very, very powerful. It’s a powerful tool just like television, like books, like movies, but it is only a tool, and we still need to use our wisdom, our critical thinking to utilize information and stories that are coming from all the sources, including AI. Now, if we talk about financial technology, the finance, over seventy percent of women have bank accounts. Only one in three actively use their accounts. So a majority of women access banking services via men in their family. Financial literacy is low, and so is technical literacy. It’s not promoted by women in any geography. And as an example, women are eight percent less likely than men to own a mobile phone, especially in low and middle-income countries. So even though computing is evolving with machine learning and generative AI, the problems that are coming with that data that we have been gathering so far, it’s still biased, and it’s high time that we take corrective measures because the way we’re gonna shape technology today, it’ll shape the behavior, the routines, the values in future, and we wanna make it good. We wanna make it sustainable. We wanna make it scalable.

Caroline Béguin: Before we can tackle the issue, we need to understand it. So what do we mean by gender bias in AI, and how does it manifest, particularly in sectors like banking and fintech, where data and algorithms increasingly shape decisions? 

Dana Lunburry: Yeah. So when we’re talking about gender bias in AI, we’re thinking about the ways in which the AI systems are producing and reproducing unequal outcomes based on gender, and this is often with unintentional consequences that’s happening in the real world. So what’s happening in financial institutions and across the industry, it’s something that we see matters very deeply because we are using AI to make decisions about so many things. So we can be thinking about credit scoring, loan approvals, insurance pricing, fraud detection, all of these things and decisions that are directly impacting people’s everyday lives and, of course, their financial futures. The problem related to this bias that we’re seeing is really starting with the data. So if AI is trained on historical financial data, it’s reflecting past discrimination. So say, for example, women historically have been granted smaller loans or have had limited access to credit, and in some countries it’s still illegal, so lots of discrimination throughout the ages in all countries really. Then because of this data and the slant that we see in the data, we know that AI is learning these patterns, and they’re replicating them, even if gender isn’t explicitly included as a variable in the dataset. So, for example, there was a well-known case a few years ago where women were being offered dramatically lower credit limits than men, even when they had a similar financial profile, and that’s really a great example, I would say, of bias showing up in a seemingly neutral algorithm. It’s also show-showing up in more subtle ways as well. Behavioral data is there, things like shopping behaviors and patterns and even browser history. These can all carry gendered assumptions that end up influencing credit decisions. So investment platforms might unknowingly, for example, cater more to a male risk profile and then leave women underserved. And if the data is used to train these systems, if that data is underrepresenting women or non-binary individuals, the models just aren’t going to perform as well for them. That’s not just a technical issue, but it’s really an inclusion issue. 

Bettina Vaccaro Carbone: I think what is interesting and important to keep in mind is that, as Dana was saying, there are two aspects in which this is gonna be impacting women and minorities in general. One is that in the decisions that are made, as we believe that going through AI is gonna take out the human bias, we’re actually leaving more space to bias based on historical data and decisions that were made previously with a different view of things, but also because there is those-Blind angles and the impact that it has on the side because as Shalien was saying, women are less likely to have mobile phone or are less likely to be active in their financial life, which means that when you take both of these aspects, you’re just increasing the risk of putting people to the side or not being inclusive enough. And once again, the risk is not having the capacity to solve today’s problems by looking at the world in an historical way instead of changing or triggering that movement. If we add another thing, because we are obviously talking about the use of financial tools, the AI is not only used within the financial tool, it’s also used for recruitment, and it’s also gonna be used for trying to train and do financial literacy. So it really impacts the whole chain of value if we don’t try to think about how we can change these kinds of things. 

Shalien Kishore: Just two cents from my side. So at SBS, we have the digital sustainability pillars of accessibility and eco-design. And accessibility is not just about disability, it’s also about how different genders, people of different age groups and digital literacy access the same systems. So for example, high-pitched voices or ethnic accents, it’s difficult for the voice recognition systems to recognize them or sometimes even there are higher error rates for women and people of color on facial recognition because the data itself, the data collection, the data curation, it was imbalanced, and that’s where we need to start solving not just from a technical perspective but from a human perspective.

Caroline Béguin: Could the way we build and run our digital tools actually make them fairer? In other words, can digital sustainability help us fight bias in AI systems? 

Valmina Prezani: I will start by re-explaining a little bit about what digital sustainability is, and then I’ll let my peers who are better in the technical part explain how we are actually using this to improve the way we work. But digital sustainability, as Shalien says, has four pillars, and at SBS we work specifically with two of them. So the four pillars are gonna be green IT, so how you make sure that your system are mindful of the impact they have on the environment, and you’re gonna be looking for improved performance but also to make sure that the conditions that you’re offering are in line with the needs. So to give you an example, a client that might not be as educated in these kinds of things will tell you like, “I need a hundred percent availability for my lending rates.” And you’re like, “Well, in reality you’re only using your lending rates once a month to recalculate your interest rate for the loans ongoing, so maybe you don’t need this information available a hundred percent of the time, twenty-four seven.” And probably some technical people will, will improve the way I explain this, but the idea is that you also need to dimension the services in line with the need of the services and not just with idea of what it should be because as of today, and it is the same as personal views, like we want Amazon to deliver things in our door in an hour, but do we really need something in an hour delivered every time? Not really. It’s just like we have grown used to this way of working. So green IT is really about making sure that we design in a way that is in line with the needs but that is mindful of reducing the impact that we have on the environment because as you know, it’s not just about CO2, it is about water stress as data actually we’re mentioning, and the impact that it will have in the communities because if we don’t have enough water, we don’t have enough food and all of those things. So that is one part. Then you have IT for green, but it’s also preventive maintenance and making sure that you update and upgrade in the right time and not just all the time ahead or after breakage because that is also having an impact in the way we work. You have accessibility, which as Shalien was saying, is not only about disability, but it’s also about making sure that our software are available in low debit communities because if you only have mobile bankings, you need to also be able to access your banking even if you only have 3G or less. And it’s also about if you’re elderly and maybe you don’t have the same access or capacity to use a phone, like how do you still get access to your banking services? And then you have your ethics pillar, which is about being mindful and thinking about how technology has an impact in the life of people and not just thinking that because it’s technology, it’s gonna be neutral or not gonna have an impact. And that is where we’re gonna be able to come into things like an AI code of ethics and how much freedom we give to the AI or how we can think about what we’re gonna feed into the models in order to make sure that they respond to what we’re expecting, and that’s where I’m gonna leave the floor to my peers who will be able to explain this even better than me.

Dana Lunburry: I can talk about this from a technology management perspective when I did my PhD on this topic, and something that kept coming up is about the role that technology plays, right? So technology actually has certain affordances, and we can think of them in this idea of the opportunities and constraints that we can design into the technologies themselves to get the outcomes we want. So for example, when we think about designing AI more inclusively, we can factor in its different diverse voices, especially women, and know that the technology will become more socially responsible and environmentally aware. And some of the things of those principles we can be thinking about when we are looking to reduce that gender bias and environmental degradation that’s being perpetuated by AI systems today, we can be thinking about three ways or three techniques. So to speak. So first of all, we can think about how we compensate? How do we change technology to compensate for the harm that we’re seeing? So we’re using AI technologies, for example, to compensate for those negative outcomes of environmental degradation and gender bias by adding something positive. So it’s about using technology to deliver something positive that’s gonna offset what’s being negatively outcomes in that situation. So for example, we may need to create data that has more gender balance than what was typically found in the data sets from the real world, right? So if we’re missing data from women, we need to start creating synthetic data, maybe gathering more data from women to actually compensate for that situation. The second principle I can mention here is transformation. So how can we fundamentally change technology, right? So the use of AI technologies to transform negative outcomes and change that into something more positive. So using the technology to at a more fundamental level to change what was that, that was negative, right? So this bias we’re talking about, the degradation of the environment, and we can design AI systems to fundamentally change the interaction between the AI systems and the users. So right now, users are actually really struggling with prompt engineering. It’s hard to craft prompts to limit the bias when it comes to gender. It’s very difficult, and the responses back also aren’t providing the transparency needed for the users to understand their data sources. They’re not understanding the interpretation of their prompts, and there’s potential for bias to be creeping in at, at any stage, and users aren’t seeing that. What is the bias? Where is it coming from? And so we can actually rethink the interaction entirely by giving users the tools that they need for prompt engineering and to enhance transparency so that they can make decisions that will combat these issues. And we can transform these systems to add both enablers and constraints that would be useful for reducing gender bias and to help improve our ecological footprint. Now, the last principle I can mention here is counteraction. This is about the use of AI technologies to counteract negative outcomes. So we’re using AI technology, for example, to prevent the occurrence of what is currently negative, so the bias again and the degradation of the environment. So imagine in this situation, advanced AI systems that sit alongside developers and flag bias and recommend inclusive data sets and auto-adjust models to reduce harm, like, kind of like having an ethical co-pilot right next to you, right? That’s gonna evolve with every line of code and every prompt that you give. So those are some examples based on some of those technology management principles. 

Shalien Kishore: And from an SBS design perspective, from an inclusive design for banking software, we can bring in more us-universal design principles into the process itself. So for example, we can eliminate the biased assumptions in user flows. We could design with multiple personas that include women entrepreneurs or gig workers or single parents and rural users. We could remove the gendered assumptions and product features, so for example, we don’t need to assume that only men are the primary bread earners or decision-makers. We can use accessible interfaces and interactions for all abilities and literacies, financial literacy and technical literacy. For example, we can use plain gender-neutral language. We can use visual aids that make it more inclusive. We provide readers, screen readers, text resizing, voice-assisted navigation. We can offer local language support, especially in the regions where women or older users may be less fluent in the default language. And then we can make gender-aware credit or risk models, so the users aren’t penalized for gender or a marital status or a caregiving role. We can incorporate alternative data sources like the mobile payments, the community group behavior to assess the creditworthiness. So AI opens up the possibilities to make things more comprehensive and more inclusive because our processing power is immense, and we can utilize that to make things more gender-neutral. 

Bettina Vaccaro Carbone: One of the things that we can also do, and picking up on what Shalien was saying before, as SBS, and that is something that we’re currently working on, is also support our customers in better understanding these kind of things because we have started this journey a couple of years ago, and although different people are at different rhythms, I think it is a unique opportunity that we have today to also have these discussions with our customers and help them be more mindful on how they can use our technology in that sense, as Shalien was saying, in trying to incorporate different ways within origination to make sure that they are more inclusive or reducing the idea that maybe the man is the sole breadwinner and things like that. So because of the role we play in our ecosystem, we also can be advisors, and we can support this growth on our customer side. And it is something that we have started with some of our customers, where we’re giving digital sustainability lessons or digital sustainability courses in order to be able to interpret and integrate these kinds of things within their business processes and thinking.

Caroline Béguin: Okay, so here is a bigger question. Can we create AI that’s not just smart and efficient but also good for people and the planet? 

Bettina Vaccaro Carbone: I think it’s not so much about energy efficiency and responsibility. I will always go back to the same thing. We need to be aware of the problem, and then we need to work toward a solution. As long as we’re using resources, as long as we’re developing software, as long as we’re consuming, we’re still gonna be using and consuming resources. But what matters is to think about the need. And for example, we do have within SBS a person who is responsible for studying the impact of AI in the environment and who we are creating a framework on how to make decisions in the way we implement AI to make sure that we also stop the models before going too far, because we do know that the further the AI model is trying to look for a better answer, the higher the impact and the cost in energy and resources. However, the improvement becomes minimal as we move along. So here are real questions about how to choose which model depending on which case we’re using and how far we want to go in the research, and all of those things are about understanding the problem, understanding our objective, and trying to put the right level of solution. So it is about being in the right amount. 

Shalien Kishore: Equilibrium. 

Bettina Vaccaro Carbone: Thank you, Shalien. 

Shalien Kishore: I would flip that question a little bit. So we should not say, “Is it possible?” We should ask the question, “How do we make it possible?” Because it’s increasingly essential to build AI systems that are both energy efficient and socially responsible. So we have to make deliberate design choices. We have to set up regulatory frameworks, and we have to have a shift in our mindset from the bigger and faster to what’s fairer and smarter. So whether we’re talking about optimizing the model architecture as Bettina just talked about or we are working into green AI training practices or even edge computing on device AI so that we save resources in the back and forth. We need to figure out solutions to make it energy efficient for sure, because we know it’s gonna go big, and it’s gonna get bigger and bigger as we move into the future. 

Dana Lunburry: Yeah, I would completely agree, Shalien. And I think too, as we think about how we reduce and remove bias in AI and the data, underlying data.We know that we need to use frameworks, we need to be auditing, we need to be retraining and retraining our models. All of these things are going to have an impact on the environment, and the situation is once we start gaining these best practices and standards, and we can apply them in efficient way, that’s really the best way to go about it, is not to say, “No, we’ve got to minimize how we’re leveraging AI models,” and actually cut out things that are gonna make it better, especially when it comes to reducing bias. But we have to be thinking smarter and working smarter so that we do have less of an impact on the environment.

Caroline Béguin: From a business point of view, why should companies actually care? Does tackling AI bias and going green really make business sense, or is it just about doing the right thing? 

Bettina Vaccaro Carbone: As of today, there’s a bigger interest for our clients in these kind of things, and there are numbers that show that this is a question that is asked more and more in RFPs and RFIs. We also know that, for example, in Europe, the CSRD, which is the global CSR regulation at the moment, is asking us to be accountable for all of the value chain, which means our customers and our suppliers. And from a business perspective, being able to deliver on these kind of things, being able to answer to this question and making sure that we have the right level of implementation in these kind of topics is gonna allow us to have at least an answer to the requirements, but also be able to have a differentiator in the type of products that we sell. We know that accessibility is a differentiator today in a lot of places, and for our customers, being able to propose these kinds of services and being able to showcase that they’re working towards this is also a differentiator towards their customers. So it creates a value creation, a positive cycle that I think it is important to keep in mind. So in a very cynical and, like, money driven way of saying the world, doing this actually makes us more efficient, more performant, so we have a bigger profit because our products are spending less money in the way they work, but we’re also getting more clients and more business. So from a business perspective, it’s very easy to justify it, and I’m sure that Shalien Kishore will add some interesting points. 

Shalien Kishore: I fully agree. So energy efficient AI models, it lowers operational costs. It increases efficiency. It’s straight away good business. It’s a leaner infrastructure. It’s the optimization storage, lower maintenance expenses. So yes, everything improves the operational costs. Plus, it’s a competitive advantage and drives innovation because the moment you make a team that’s focused on ethical AI and sustainability in the long run, you unlock creativity, you unlock future-proofing of solutions, and you get ahead essentially with diverse and inclusive teams. There was a study with BCG. It shows a nineteen percent innovation boost in companies with above average levels of diversity. So yes, it makes business sense. It makes innovation. It gives you a competitive advantage.

Caroline Béguin: So what now? What do we do with all this? If you’re a developer, a designer, or making decisions that shape tech, how can you help build AI that is not just smarter but also fairer and more sustainable? 

Shalien Kishore: So yes, it’s a shared responsibility, and as people who are designing and developing AI solutions, we need to ensure that our systems are fair, transparent, and unbiased. One, we need to use diverse teams and representative data sets. We need to regularly conduct audits. We need to implement explainable AI so that we can cross-question it and understand how the decisions are being made. And we need to advocate for diversity in AI teams so that we’re reducing blind spots. 

Dana Lunburry: Yeah, and AI certainly has a role, too. Exactly what Shalien is saying. And when we add on a layer of AI to do AI governance, for example, how can the technology itself be used to all of these objectives that we’ve been talking about throughout this? I think that would be extremely powerful. 

Bettina Vaccaro Carbone: And I will just come back to what Shalien was saying in the beginning, which is, like, we need to keep thinking that this is a tool. And as a tool, it is something that we need to govern. So we need to implement the framework. We need to have the right people working on it, and we need to make sure that we do the checks that make us follow up and make sure that things are working the proper way. So it’s really also our responsibility to make sure that this answers our needs. 

Caroline Béguin: So to wrap things up, we’ve seen that building better AI isn’t just a nice idea. It’s something we have to do. And it’s doable, so that’s good news, but it’s just a start. So according to you, what will it take to keep pushing for technology that serves everyone equitably and responsibly? 

Shalien Kishore: So whether we like it or not, AI is going to shape the next generation’s worldview. If it is biased, so is our future. So we have to tackle it, and we have to tackle it now. So building a world where people see limitless possibilities regardless of their gender, caste, creed, color, anything, we need to make a fairer world because it’s not just a technical flaw. It’s a societal issue. And the question is not whether AI can be fair. It’s whether we have the will to make it so. 

Dana Lunburry: And I think this issue around governance and making sure that IT systems and AI systems are governed in the right way to get the right outcomes, as Shalien’s saying, that we do have a fair, fair world and equitable outcomes. And all of these things are so critical and important. We want to see that governance standards continue to rise and that they get implemented as best practice in every industry. So that’s something that we’re already seeing a lot of change on, which is great, but it’s something that we have to keep pushing for and keep advocating for, and there’s a role for everyone to play in this because AI touches everyone’s lives. So if all of us are responsible for getting our feedback back to companies, making sure we get the transparency we need, making sure that as systems progress, we’re being critical. Are we being treated fair? Are we pushing to create the world that we wanna see? 

Bettina Vaccaro Carbone: I think to, to highlight, Shalien and Dana, I think we are all responsible for this. So one of the aspects we haven’t mentioned because we’re talking really about the company, but it is about how everybody needs to be educated in this and how we need to try to make sure that every person, whether it’s yourself or family, our kids, our community, is aware that this is something we are all responsible for and that we keep advocating and expecting this from our companies, our governments, the schools where we are, everywhere. We need to be mindful of the role we play in this game because if we are not asking the questions, if we are not being there, if we are not trying to change things and voice out the need that we have to make this an important topic and have it on the table, it just get thrown away with other things. And in a very cynical world where we are driven by financial market indicators, we also need internally to be able to continue proving that this brings value and that it makes sense. So that means knowing what we’re following. It means having KPIs that we can showcase and being able to track these evolutions in order to make sure that this is taken seriously at every level, even though already being there and voicing it out is an important part of the journey. 

Caroline: Thank you so much, Bettina, Shalien and Dana, for sharing your insights. It’s been a very thoughtful conversation. 

Shalien Kishore: Thank you, Caroline. 

Dana Lunburry: Yeah. Thank you so much 

Bettina Vaccaro Carbone: It’s always so fun to be with you guys talking about this amazing topic. 

Shalien Kishore: Always.