Alexis
Hi everyone. Welcome to Talk Tech with Data Dave. I’m Alexis, your host of this podcast. I hope that you are ready for the second part of this Data Lakes versus Data Warehouses Explained episode. We left off after explaining what a data lake and a data warehouse are, with Dave trying to tell me that sometimes he puts things in a data lake and then puts little data warehouses inside of a lake, and then said, “Does that make sense?”
Well, you can take a guess at what my answer is.
Data Dave
I actually define data lake bigger. Right? So, I will put areas of warehouses into my data lakes. I’ll have some very, very structured data and some semi-structured data in my data lake and use them in conjunction with each other, and that becomes part of the broader data lake environment.
Does that make sense?
Alexis
No, but probably because in my brain, a data warehouse is a structure and a data lake is a lake. And so, there’s this picture of this structure inside this lake, and I’m just like, why would you do that? How do those things work together? And, and like I have to remember that that’s really not how it works.
Data Dave
Yeah. So think of your lake, which is all your data, and then put a bunch of islands in it with solid structures on it.
The islands are solid structures, data warehouses, or data marts, small data warehouses in a lake of unstructured data.
Alexis
Uh huh.
Data Dave
Does that make sense?
Alexis
And then those things can actually help all of the other things; random stuff.
Data Dave
Exactly, exactly. Those things give you pivot points and gravity points where all the data that’s in a mess starts to come together to be rationalized through these lenses that you put on it. With the island, with the little castles on an island. Think of a little island with a castle in the lake.
Alexis
Yeah.
Data Dave
Or a fishing shack on the lake. It’s, “I’m doing something with this lake. So the data is flowing through the lake and being used through the island, but can actually be put to use.”
Alexis
Because there’s other things there to support it, to help. Right?
Data Dave
Exactly. I’m putting more structure on it, I’m applying structure to it, but it’s late-binding structure as opposed to early-binding structure. I didn’t have to define the structure before I could put it in the lake. I put it in the lake, and then I defined the structure for it.
Alexis
Okay.
Data Dave
This means that a data lake becomes very useful for things like data science, machine learning, AI processing, and that sort of thing. Where you’ve got all the data that is relevant to a subject in the same place, but it’s not highly structured.
So, I want to go and find something interesting, this data.
Alexis
That’s what we all want, right? We always want to find something interesting in the data.
Data Dave
Exactly. I want to find something interesting in this data. Now, it’s kind of, “Well, I’ve got all the data, and I’ve got the subjects for this data, but what I haven’t got is all the structure on this data. So I’m going to investigate, I’m going to explore the data in my lake to find the different structures of this data, find the data that is relevant, and start to do something interesting.” And I might create another island or I might pull all the relevant data for my question into what I call a data pond.
Alexis
So we got a lake.
We got a pond.
We’ve got warehouses that you are now calling islands.
Data Dave
That’s right.
Alexis
Okay. Okay, tell me about a data pond, Dave.
Data Dave
A data pond is just a small data lake.
Alexis
And is it part of the lake?
Data Dave
Well, it might be.
You might say, “I’m going to get all the data that’s relevant to my sales in New Jersey, and I’m going to pull it all into a pond because I want to do some deep analytics on that. I want to feed all that to an AI engine. I’m going to do an experiment on this subset of my data.” So rather than give myself this huge lake of data, I’m going to take a subset of that, which I would call a pond.
Alexis
So, just a smaller version.
Data Dave
A small lake.
Alexis
Okay, a small lake. Got you, got you.
Data Dave
So, all this is giving you knowledge of your lake. Your lake is all your data. And you could argue, you can discuss, whether a lake is multiple small ponds or whether a pond is a subset of a lake. We can have that discussion over beer. And that’s purely semantics. Right? You can have that conversation. And you should have structure on your lake. You should have some governance on your lake. You should know where to find things on your lake. Which is why I cut lakes into ponds, because then I’ve got a pond that has this subject in it, a pond that has that subject in it, and it’s all the same lake, but it’s multiple connected lakes, or it’s a string of lakes. Right?
Alexis
So that there’s structure on it. So that when it’s time to go fetch things from the lake, it makes more sense to know where to find it.
Data Dave
Where to find something, yes.
One of the things you’ve got to be super careful of is, when you create these things, are you creating just another place to lose stuff?
We’ve all got that drawer in the kitchen or that closet which is just…Yes, it’s the junk drawer. We throw everything in it. We pretend we’re organizing ourselves, but really it’s just another place to lose stuff.
Alexis
Yep.
Data Dave
That’s what my garage is. It’s just another place to lose stuff.
So, we’ve got to be careful when we create these lakes and these warehouses and things, that we do them with purpose, that we do them with knowledge, and we don’t create another place to lose stuff. I’ve seen people create data lakes, and then they throw everything in it, and then the lake becomes a liability because nobody knows what’s in it.
Alexis
That was going to be one of my questions.
Data Dave
Then you can’t give everybody access to it because, “I don’t know what they might find. Now I’ve got to lock people out of my lake, which means it’s counterproductive.”
Alexis
To live by the least privilege. If you’ve got everything in your lake, you can’t give everybody access to everything.
So now, nobody can have access to the lake.
Data Dave
So, what we often do is create zones in our lake. So, you’ve got a highly secure zone, a financial zone, a product zone. Well, those are ponds really.
Alexis
Those are Dave ponds.
Data Dave
Yeah, subject area. So now, you’ve got some structure, you’ve got some governance, you’ve got position on your lake, otherwise your lake, to use a classic term, your lake turns into a swamp. Right?
I’ve got my lake, but it’s awful. It’s dirty, it’s polluted, it’s got out-of-date stuff, and it’s got all kinds of crap in it. It becomes a swamp.
Alexis
Yeah.
Data Dave
So, we want to keep our lake clean or cleansed. It’s not as clean as a warehouse, it’s not as clean as a report or whatever, but it’s cleaner than just garbage.
Alexis
So, we’ve kind of already talked about the magic of a data warehouse. It’s the- I know where things are, I can get there quickly, I can read them quickly. That’s what it’s there for.
The magic of a data lake is the ability to add structure, kind of, to unstructured and semi-structured data after the fact.
Data Dave
Yes. And you made some very good points there. It’s to add structure to semi-structured and unstructured data after the fact. The question you didn’t go to is why?
Alexis
That’s a good point.
Data Dave
Why would you do that? You do that because the big thing about a data lake is you’re exploring the unknown.
In a data warehouse, everything is known because you built it with structure and with purpose.
In a data lake, you built it without knowledge.
Alexis
Right.
Data Dave
It became a collection zone with some governance on it to keep it cleansed, not necessarily structure. Hopefully, that means I can explore it. That means I can find interesting answers to interesting questions, and I can apply new questions to my data lake that I can’t apply to my data warehouse because my data warehouse is tuned to the questions that I designed it for.
Alexis
Right.
Data Dave
Whereas my data lake isn’t tuned to that. It’s just tuned for storing and applying structure to data.
So, my data scientist can now say, “I want to see all the data in the world about New York City that we know about in our organization, or about mountains, or about whatever.” Now I can explore. “Well, I’ve got an interesting question. I’ve got a question that says demographics. Where people live in New York City, my street.” Does that exist? Not in a structured way. People don’t necessarily ask that question.
Alexis
Right.
Data Dave
But I’ve probably got data that describes it. So, I’m exploring questions that are not primary questions by adding structure to my lake. I’m starting to look at the data, I’m exploring the data. I’m starting to say, “Well, this is relevant to this answer, it’s relevant to this question, therefore it can be part of the answer.” So now I can start to do interesting analysis, and I might find something that’s interesting enough that I now add structure to and put into a warehouse. And now I can make it available to more people.
Alexis
Yes.
Data Dave
So, I can explore my data lake in a much richer manner. And I’ve got semi-structured data.
So, I’ve got all these documents that are in there. Well, why don’t I feed those to my AI engine? They’re all on the same subject. So now, let me have AI read those and extract my answer for me until I can start to say, “Here’s this answer to this question. That is a very specific question.”
Alexis
Yes.
Data Dave
So you use it as a cleansing area so you can parse the data through, cleanse it, clean it, and pass it up to my warehouses. Pass it through that. My warehouses are answering particular questions. My data lake is the supporting information that’s feeding through that, and I’m using it for data science and analytics to lead into answering questions.
The way I design analytics is, “What is the question you’re asking?” I would then ask you, “For your organization, where does the answer live? What data do you have that you think contains the answer?”
You might say, okay, I’m asking this question. Your question might be famous mathematicians from the 1800s. “I want to know the discoveries from famous mathematicians in the 1800s.” Just a random question, okay, why did I choose that question? I don’t know. Random question. So, I’d go to a library. You go to a library and you say, “This is in this section of the library. The answer is in this section of the library. I don’t know what the answer is, but it’s in the section of the library.”
So now, give me an AI engine to go and read all the documents in that section of the library and give me the structured data. Give me a pipeline of data that says, Now, if I want to know interesting discoveries by mathematicians for any point in history, 10 year period, whatever it might be, I can now answer that question diligently by pulling that data from my lake into a warehouse that describes this data. And it becomes very, very efficient.
I’m starting with a question, I’m looking at, where does my answer live? And then I’m doing the analysis and building a refinement process to pull the answer to the question out of the data that contains that answer.
Alexis
It sounds to me like these two ideas, if you will, often work very well hand in hand. They should and maybe aren’t necessarily required to be used hand in hand, but in an ideal world, would be almost a streamline. It would be here. I’m going to work my unstructured data through a data lake so that I can help it be more structured so that I can put it into a warehouse so that I can easily answer the questions that I know I’m going to be asking over and over again.
Data Dave
Right, Exactly.
Alexis
And that’s the filter that everything you’d be working on would constantly be running through.
Data Dave
Exactly. The warehouse contains the answers to the questions that you ask all the time. The data lake contains all the data that contains all the answers. And you might not put everything in a warehouse.
Alexis
Right.
I do realize, though, Dave, that I forgot to say early on, and now we’re at the end of the podcast, and I’m going to say it anyway. You probably noticed that there’s no video version of the podcast right now. Dave and I are trying this audio-only thing just to see how it goes, see what we feel about it, see how everyone feels about it. So, if you like it or don’t like it, let us know in the comments. We’d love to hear some feedback on it.
But I think we kind of answered the question. I think I have a much better understanding of the two concepts here. Dave, before we go, any last thoughts or you think we kind of hit everything on the head?
Data Dave
No, I think we hit everything.
Alexis
Well, thank you, everyone, so much for joining us for another episode of Talk Tech with Data Dave. If you have a question for Data Dave, please send it to us at talktech@d3clarity.com or talk to Dave or myself on LinkedIn. We would love to chat with you on any of those places. Other than that, Dave, thank you again so much, and everyone, we’re excited to be back.
Data Dave
Thank you.
