Alexis
Hi everyone. Welcome to another episode of Talk Tech with Data Dave. I am Alexis, your host of this episode and I’m here today with Data Dave. Yes, our summer has come to an end, and Dave is back joining me. Hey Dave, welcome back to the normal podcast.
Data Dave
Yes, thank you, Alexis. I’m glad to be back.
Alexis
I am super happy to have you here. Now you get to hear me say, listeners out there, we are always looking for listener questions. Feel free to reach out to us at TalkTech@D3Clarity.com, right on the D3Clarity website or connect with D3Clarity or Dave or myself on LinkedIn and ask us questions there. We’d love to answer listener questions on the podcast.
Data Dave
Absolutely.
Alexis
But today, Dave, we want to answer a bunch of questions. So let’s fill the listeners in on a little bit of context.
Back a couple months ago, you did a presentation with DATAVERSITY’s Data Architecture Online.
Data Dave
Yes, correct. I did.
Alexis
And that presentation was called Beyond the Algorithm.
Data Dave
Yes, indeed it was. Yes, it was.
Alexis
And it was about kind of the mesh of master data management and AI. The mesh of master data management and machine learning.
Data Dave
Right, right. Quality of data and machine learning and AI and that sort of thing. And how does quality of data affect your machine learning and strategies of that kind?
Alexis
We have like white papers and blogs about that, and I know we’ve talked about that a bunch of times on the podcast before. So, Dave had so many people join his session and so many people ask questions that we didn’t get to all the questions during the session. So, I made note of those questions, and now that Dave is back on the podcast, I thought we would do like a rapid-fire Q&A of the questions that we didn’t get to during the session. That way the people who were there who have those questions kind of sitting out there can get their questions answered.
Dave, I think we’ve got five to try to get through today. We’ll get through as many as we can. Are you ready to go?
Data Dave
I am ready to go.
Alexis
Okay, first question.
How do you manage modern data architecture for unstructured data and for AI and ML?
Data Dave
So, the first question is, what kind of architecture are we talking about? Are we talking about a storage architecture, or are we talking about a logical data architecture from a storage architecture? There’s a whole bunch of ways that you can store unstructured data and make it available for AI and ML, and have it read and parsed and that kind of thing. So, go for either a cloud and a document management system, etc. So, you keep your documents fresh, keep them structured, or keep them in a structured storage context. I’m not going to go into the exact technologies, but lay them out in a reasonable library so that you can at least find them and point your crawlers at them from a content perspective. Now, you’re talking about the extraction of metadata from all that unstructured content and building a knowledge graph across the top of that to allow your AI and ML to traverse what piece of content means what.
So, I’ve probably just answered that question, which is: do the extraction of the topics that you need. So, use another ML or AI type engine to extract from video clips or particular images or content of interest from the photograph, from the document, from whatever. So, extract the content that is interesting. And then build a knowledge graph that connects and tags all this content together to give you a content knowledge graph that then you can feed to your AI model to start saying, “How do we traverse this knowledge graph to answer those questions?”
Alexis
Short answer that I heard was- do your best to try to structure it, and then sometimes you can actually use AI and ML to structure it, and then you can feed that into the AI and ML.
Next question, what are the correction steps you tell your clients to make to a gen AI system that has been trained on bad data? So, we talk about all the time, garbage in, garbage out. If we trained a gen AI system on bad data, what are the correction steps to take to try to fix it?
Data Dave
If you’ve got seriously bad data, you actually want to remove the data. You want to retrain it from scratch. Yeah, so it doesn’t have knowledge of the bad data, so it will never give you a bad answer again. But that’s kind of another relief set. You’re doing a release that has trained it on a better set of data, therefore it will give you different answers. It is very difficult if you’ve got a large language model or gen AI structure to remove bad answers from it. It never really forgets bad answers.
So, while you might be able to teach it to say, “Don’t ever use this piece of content ever again in your decision making,” you’re actually telling it to use that piece of content in the decision making because you’re telling it not to use it. So you’ve got to be extremely careful with that kind of corrective action. You’re probably better off if you can retrain it with an absence of the particularly bad data.
Alexis
That was what I would think would be the best step.
Data Dave
Doing a version two.
Alexis
Yeah, I know that we’re doing a project, or we just finished a project where we trained an LLM and sometimes it was giving us bad answers, and ultimately we did decide that it was because the input was not great, and we were able to just swap out the input, and then all of a sudden it was giving us better answers.
Data Dave
Exactly.
Alexis
It wasn’t that it was finding the wrong anwers, it was the answers it was finding weren’t really the best answers to find. And so, we were training it on not great stuff. And when we fixed what we were training it on, it was giving us a lot better results.
Data Dave
Right. If you’re doing a narrow knowledge base, like really using the LLM for reading comprehension.
Alexis
Yeah.
Data Dave
“I want you to read this article, this set of books, and then I’m going to ask you questions about them.”
Alexis
And that’s more or less what this was. Yeah.
Data Dave
The set of books, the set of content is relatively static and you’re giving it to them relatively frequently. So, because you’re giving it to them relatively frequently, to drop three books from the library and say “Tomorrow, I’m not going to ask you to read those three books.”
You can’t read those three books anymore because I haven’t given them to you. I’ve taken them out of the library that you’re allowed to look in because it is only going to answer within the carte of information it has.
Alexis
So yeah, the next question is a totally different point of view, and I really, really kind of like it. How do you deal with the database-agnostic people who willy-nilly switched databases and now want to change everything in JSON?
Data Dave
Well, that’s an interesting question. How do you deal with them?
Carefully persuade them otherwise. With a large hammer? No.
Alexis
A bottle of whiskey. Sometimes I persuade people on our team like that.
Data Dave
Exactly. I can be persuaded by that.
There’s no really good way. What you’re really asking is – people who jump technologies and technology isn’t the answer. The context is the answer more than the technology. Whether the data is in JSON or SQL Server or Access or whatever. And yes, some people do still use Access. It’s really the nature of the data.
You want to pull them back to the data and not the structure of the data. As a Data scientist, you want it in a structured form as much as possible, but you’re also going to change that structure yourself, because you’re probably going to do some encodings on it and some analysis on it that pulls apart some of the factors and does some different things on it as a data scientist, as a data engineer. So, abstract yourself from the format of the data as well.
I do a lot of data science with a lot of databases, as you know, all the time. So, my point is, I’m just happy if somebody gives me data in a form I can read, because I get so much that’s not in a form that I can read easily. So, to get it in a database or get it in JSON is something to be celebrated. If they change from this to that, then, okay, it’s annoying, but it’s not the end of the world.
The thing that usually rubs me up the wrong way is when I ask for data and then people tell me what they think I want to know, rather than what I actually want to know, right? Or they don’t give me enough data. They give me a copy of a report in a PDF, and that takes a while to understand, but then it’s a report. So who knows what filters, what columns were eliminated, whatever was done to it. I really want the raw data, because I can work with that much, much better. And then I can filter it myself, because the number of times I’ve fallen into asking people for data and then they give me what they think I want to know rather than what I need to know.
Alexis
Oh, you and I did that one time. I built a report for the senior team here, and I was building tables and graphs based on what Preston asked me to build tables and graphs on. And I hopped on a call with you, asking you if you felt okay with it, and you were like, “I don’t want all this crap, Alexis. Give me the raw numbers. I’ll make my own tables and graphs.”
And like, I locked it down. And I was like, “All right, Dave, here’s the password. You can just do what you want with it.”
Data Dave
But that happens all the time, right?
I’m doing a project with 87,000 rows of data, at the moment. And it’s kind of, they wanted to just give me 6,000 because they wanted to know that. But there’s stuff hidden in the whole 87,000. I want all of it. They’re asking me about 30 columns. Well, it’s 60 columns wide, and I’m using the whole 60 and the whole 87,000 to give them back the knowledge. “How did you find that?”
“Well, if I’d just gone with the data you wanted to give me, I wouldn’t be able to. But I wanted the whole data set.”
Alexis
So, the best answer is, be nice and try to lean into the data and try to not worry so much about where the data is or what the data’s in.
Data Dave
Exactly. And what I often do is just, for my own peace of mind, I’ve always got a number of databases in the background, so I will often take the data as given and before I do anything with it, load it into a database for structured data. So, my load and my analytics are separated.
Alexis
Okay.
Data Dave
So, I read whatever format I’m given and load it into a database, and from that, I then do my analytics. I don’t analyze the raw form that way. My analytics are not tied to the format of the data that it’s given to me. And I only have to rewrite a Python script to load JSON vs SQL vs CSV vs Excel vs whatever.
Alexis
Fascinating. That’s a really great take on it.
Data Dave
So, I separate those two pieces and do all the analysis off of either a SQL Server or Postgres database.
Alexis
The next question had a little bit of context behind it. We were talking about CDEs, critical data elements, and we talked about this on the podcast.
Back when we were talking about CDEs, you had said something in the conversation about considering the fact that CDEs are specific to the parts of the business. And when you and I talked about it in the podcast, you had said, you know, my CDEs as the person who runs the invoice maybe aren’t the same CDEs as you as the person who fills out the client information.
There were some disconnects based on the different parts of the business, and that’s okay. That was what you were getting at in the presentation. Okay, so here’s the question. CDEs have historically been a more formal designation, but changing them to a business perspective makes governing CDEs a little bit more difficult. What are your suggestions?
Data Dave
Abstraction is my one-word answer, right? Not helpful for anybody.
Alexis
Not helpful to me.
Data Dave
Not helpful to anybody. So let’s talk about that for a minute. So, when I say that a CDE or CDEs are specific to a business process. So CDEs have purpose. They are relevant for this purpose. Now, you can collect a bunch of them and collect them for purpose and rationalize that. But the point here is that your collection of CDEs facilitates somebody making a set of decisions or doing something. You want to send an invoice therefore, you need X, Y and Z, three pieces of information. You need the customer, you need the amount. You need whatever. Let’s go with two. You need the customer, you need an amount. That’s all you need to send an invoice. Really?
Alexis
Okay, we’ll go with that.
Data Dave
Yeah, you should need a little more. But like what? And tax and all sorts of other things. But you need a customer. Need an amount. Now, those CDEs at the business level are probably called customer and invoice amount.
When you get into a database in different parts of the database, they’re going to be specific. So, you do have to track down from the business context of what the CDE in the logical term should be and what they actually are at the physical layer. So, if you’re going to measure whether you have a good customer everywhere, then you’ve got to get to which database field am I dealing with and which field in my data is the amount that’s going to drive my invoice. You have to do that cascade and you should be looking at the synonyms for that CDE within other parts of the organization. So, you end up with this cascade across where there are synonyms with that. Now you can start to put your data quality rules in place at the physical layer versus the logical layer or the business layer.
Alexis
So, if we are doing them from a business perspective instead of formally at the higher level, just recognizing where the business and the higher level connect and then matching them that way, more or less, and applying the governance that way versus simply saying, “Okay, I have this one, I have to govern this one. I have this one, I have to govern this one.” But instead saying, “This is here, this is here. Governance matches both.”
Data Dave
Exactly. The governance has to cascade across the set. Now, another way of looking at it is to say that we’re basically lifting the CDE up to the business context and the business language, and that becomes very hard to measure. So, another way of looking at it is to say the logical tier of the business tier, it’s this. And evidence of violation of that CDE is here in my database. You look for evidence of failure at the physical layers. So, you say, I must have a customer to do my invoice. Yeah, that’s your CDE. I must have a customer. That’s my critical data element. Which customer for invoicing? If I’ve got that, at the physical layer I’m looking for, where is customer as it relates to invoicing. And then I’m measuring and analyzing the failure States of that which would be coming to you and saying, how many times this month were you unable to send an invoice without taking extreme measures to find the customer?
Alexis
Right.
Data Dave
If you had to spend time finding the customer in order to send the invoice because the customer wasn’t in the right place in the database, then that’s now a data quality failure point.
Alexis
Right.
Data Dave
And can be measured. Now, you can correct it, you can fix it, you can do any number of things, but you want to know when it wasn’t at your fingertips.
The point about a CDE is: this is critical data for this purpose. Therefore, the absence of this data causes you not to be able to perform your task. Your purpose, your critical data elements at the physical level become the evidence of failure at the logical level.
Alexis
Sounds a lot like a presentation that you and I are recording right now that I’m editing.
It’s all coming back to me now.
Data Dave
It’s all starting to make sense.
Alexis
It’s all. It’s all connected. Okay, we have two more questions. We’re going to try to power through these. Dave, we got this. All right. This one’s an easy one. I’m going to answer it. Ready? How important is Master Data in machine learning?
Very.
Okay, next question.
Go ahead. If you want to put in your two cents on that one, you can do that one too.
Data Dave
Well, you’re absolutely right. Very is a good answer. It really comes back to, as a saying in a number of presentations, right- your Master Data describes the world in which you live. If your Master Data isn’t good and you put an AI engine on top of it, then it’s not going to reflect the world that you live in very well.
Alexis
Right.
Data Dave
If you want your AI engine to reflect the world in which you live, you have to properly describe the world to the AI engine.
Alexis
Yeah.
Data Dave
So the answer very is very good. Now, it all depends what you’re asking it. You don’t have to load the product domain into something that’s answering about customers or different things. You don’t need all of it. Let’s not boil the ocean here. Let’s do what is pragmatic, right?
Alexis
Mastering the data that you need for it. The fit for purpose.
Data Dave
Exactly.
Alexis
The data that’s fit for the purpose. And I think you were talking about that in the time that this question popped up. Understanding that there’s certain data for certain purposes, and that’s what you’re going to be able to make sure that you’re understanding, that’s mastered for your machine learning, for your AI. And then this person’s basically like, “We are leading an MDM charge in our company, which we’ve never done before. And I need to know if this is backup that I can use. Can I go to my bosses and say, hey, if you want to use AI and ML Master Data Management’s important?” Everybody in the chat was like, “Yes, absolutely, send them to me, I’ll tell them. YES!”
Data Dave
No, absolutely, absolutely.
It’s vitally important, but you’ve got to be careful with it as well. If you’re an insurance company and you’re talking about customers, there’s a difference between somebody filing a claim and selling something as an upsell to somebody. So, what are you really trying to do? And what does mastering mean for this context?
Alexis
Yeah, fit for purpose.
Okay, last question.
How do you measure that your data is good enough?
So again, some context to the question. I don’t know how we got here, but you had talked about cleaning your data, your data quality, and it was understanding that at some point your data becomes fit for purpose and it becomes good enough that you can get correct information out of it, that you have working data and it’s good enough for you to move forward and you don’t need to, as you just said, boil the ocean. You don’t need to make it perfect. There’s a point where you can move forward with it. But this person made a good point and said, how do you measure that your data is good enough.
Unless it’s in an obvious flow, you don’t know what you don’t know. Is there a systematic or methodical way of surfacing and revealing flaws in your data?
Data Dave
This is a fairly in-depth question as well. So, the fundamentals of the question are: how do you measure the quality of your data, and is it fit for purpose?
Alexis
How do you measure that it’s good enough.
Data Dave
Yes, good enough. And I’m going to come back to the fit for purpose thing, which is, what purpose are you putting the data for? Right. What are you using the data for? You can certainly measure how complete the data or how accurate the data is against the ideal of collection. You can go back to say we’re collecting it here and we’re collecting it like this to describe this event.
Therefore, the event should be within five minutes of the event occurring. It should be within 200 yards of where the event took place. It should have exactly the right people in it. You can do that. But that doesn’t talk about the purpose for which you’re using the data. So, that’s the collection side of it. Is the data as good as it should be based off of the collection? Is there failure in the collection which you can fix to a certain extent.
The other point is when there is a flow around it, and I know he expressly said when there’s not a flow, when there is a flow around it, you can look for friction in the process. And in the flow, as we were talking about with the invoices before, if you do not have a customer, then that is a critical data element that is missing. There is friction, therefore in the process. You can measure that friction. Therefore you can measure the quality of the data against that friction.
When do you know it’s good enough? Well, you know it’s good enough when Alexis stops complaining about invoicing.
When the friction is low enough that it is no longer a business critical situation. Don’t guild the lily. Don’t go for making it perfect. If you can live with three failures a week, then you know you’ve got three failures a week. If you write, at what point do you say this is good enough? For now, I’m going to go somewhere else because there’s more money to be made and more of a benefit to be had if I fix that process rather than guild this process.
So, there is no strict way of doing it. The real way to look at that is if you look at your Lean Six Sigma elimination of waste structure, and you start to measure your processes and your process points, then you start to say, “I am within tolerance on this process, therefore I’m going to pay attention to that process.”
Alexis
Now, everything that you’re just saying, and maybe I’m out of touch here, but everything that you’re just saying still sounds like we’re talking about a flow.
Data Dave
Well, I haven’t yet found a construct that says this data arbitrarily is good or bad.
Right. I’ll give you an example. I’m doing a project at the moment where we’re looking at oil and gas equipment. The data is great, the data is collected, it’s highly accurate. It’s very good for reporting to senior management on the state of equipment.
But we’re now trying to use it to analyze potential failure states.
So, we’re putting it through a different set of analytics, which means that we’re starting to see this data has some significant gaps for this purpose.
So, we’ve got to augment it. The collection set is really good, but from an analytic point of view, it starts to fall down because we’re putting it to a new purpose. So, you can’t necessarily say this data is good. You can only say it’s good for the purpose that I’m using it. The data that I collected is very good. In other words, it describes what I collected.
That doesn’t mean it describes the analytics you want to do in the future.
Alexis
Right. So, the part of the question that was, is there a systematic or methodical way of surfacing and revealing flaws? Your argument here is – test it and see where your flaws are.
Data Dave
Exactly. There’s no way to know all the things that a piece of data or the set of data might be used for. So, as you come up with them, why can’t it answer these questions?
Alexis
Yeah, I think this is the perfect question to end the podcast on. And I’m happy because we hit all five, Dave, because where we just landed was, I don’t know that there’s really a systematic way of doing it. I think it’s worth just testing. But maybe there is a listener out there who does know of a systematic or methodical …first try… way of doing it. And listeners out there, if you have an answer to that question, we would love to hear it. Throw it in one of the comments.
So, give us your answers.
Data Dave
My argument is, testing it and knowing can be methodical.
Alexis
I mean, by definition, I would argue that testing to find out is methodical.
Data Dave
Right.
Alexis
That’s what testing is.
Data Dave
So, from a methodical point of view, the way I would do this is, I often draw the data model of the question, decompose the question into nouns and verbs, and draw the data model of the question that says, I am asking this question. This question is about customers or whatever, equipment. Therefore there’s a noun called equipment. There has to be a noun about equipment and there has to be a verb that is something about that equipment that is joining it to another noun, etc. You can decompose the question grammatically into the nouns and verbs that are about the question, and then decompose the data set that you have. You’ve got the data model of the data set you have. So, draw the same data model. If they are mappable, if they have the same basic nouns and verbs or synonyms, then you can answer that question with that data set.
If they do not have the same or synonyms of the same nouns and verbs, you fundamentally cannot answer that question with that data set.
Alexis
Yeah.
Data Dave
So if you want to do it without the experimenting or the cost of trial and error, there are approaches like that which will say, here’s the data model of my question, and here’s the data model of my answer.
Alexis
All right, Dave, we got all five questions. I think we did a good job. Hopefully that podcast isn’t too long. Listeners out there, if you have more questions, send them to us at talktech@d3clarity.com. We’d love to answer your questions on the podcast. Dave, thank you so much for answering the questions, and welcome back after your summer off.
Data Dave
Thank you. Thank you very much. Great to see you again. Not that I haven’t seen you this summer, but yeah.
Alexis
I mean, I guess you’ve been on some of the episodes, but it’s okay. It’s okay.
