Alexis
Hello, everyone. Welcome to Talk Tech with Data Dave. I am Alexis, your host of this podcast, and I am excited to be back at Talk Tech with Data Dave with none other than Data Dave. Dave, how are you today?
Data Dave
I’m very well, Alexis. Great to be back. Sorry, we’ve been remiss in getting together. It’s been a long time. It’s been a while.
Alexis
Yeah, it has been. We had to take a little podcast break; a little hiatus, if you will. Man, there was just like a lot of stuff going on. The podcast was kind of doing its work, and you and I ended up with a lot of work we had to get done there at the end of the year. So, we took a little podcasting break to frankly – like I said during the summer quite a bit – do our real jobs.
But I’m excited to be back behind the microphone because we still have a lot of questions to get answered.
The question today, we might as well jump right into it. Is a question right up your alley, I’m sure, because, well, frankly, the word data is in the question a lot. So, I’m just going to lay it out there, and good luck. Data lakes versus data warehouses.
Data Dave
Yes.
Alexis
What’s the difference? I know I’ve used them interchangeably, so I’m quite interested in knowing the difference between the two.
Data Dave
Okay, these terms have got a little bit muddied in recent years.
A data warehouse traditionally is a structured analytic database used for analytic-type queries as opposed to OLTP-type queries. OLTP is online transaction processing.
Alexis
There we go. You knew I was going to ask.
Data Dave
This is your normal application structure, which is inserts, updates, deletes, classic data kind of stuff. Placing orders, adding customers, that sort of thing.
A data warehouse is an optimized database for doing things like analytics. To doing financial analytics, “Show me all my sales rolled up by customer unit or product unit or whatever.” So, it’s a different style of query, different style of report. So, when you create an order, you go through a process of, “This person has placed this order, and I know all the details first. I know who the customer is, I know what order they’re placing, I know what products there are.”
Everything’s pretty structured in that way. From a data warehousing perspective, you’re looking at analytic-type queries that say, “I don’t know any of that information. I’m starting from the bottom. I’m starting with, I want to see the total value of all the orders per customer in this quarter.” That’s a different kind of request.
Alexis
It’s a different kind of request from what?
Data Dave
From the user. So, a normal database is saying, “I want to create this order,” but the structure is often very slow to say, “Okay, I’ve got all these orders now, which customers placed them.” It’s a reverse structure. So, a data warehouse is often restructured to support analytic-type queries. It’s very structured and put together to support these kind of analytic-type queries.
Alexis
So, for the layman out there who’s trying to remember what a data warehouse is versus a database versus a data lake, easy thing to remember is a warehouse is a building, and a building is very structured. That is, a data warehouse is very structured.
Data Dave
A data warehouse is very structured, and it is different from a database in the way that you access things. Okay. The idea is, in a classic application database, I’m creating data. In a warehouse, I’m not necessarily creating data. I’m reading data.
So, I’m fetching the right data, the right answer to my question, very quickly.
Alexis
Because of the way it’s structured.
Data Dave
Exactly.
Alexis
Got it.
Data Dave
If you think about a distribution center, for a minute, for retail. I create my products in one way, and then I put them in a distribution center. When I’m coming to get them, I want to find them very, very quickly.
They might take longer to put in, but I want to get them out really quickly.
Alexis
Right.
Data Dave
The normal application is optimized for creating data to transact with, and then my data warehouse structure is optimized for retrieving the answer to questions.
So, I’ve restructured the data, still very structured, to read it quickly.
Alexis
Got it. Got it.
Data Dave
So often, your data warehouse is an optimized system. When you talk about a data warehouse system, like a Redshift or a Snowflake, they are optimized for that read processing versus a traditional database, which is optimized for read and write. And then you can do data warehousing in a traditional database. You just structure the data a little bit differently. You structure the way you lay out the data a little differently to help the query engine find it quickly.
So, your data warehouse, you can build a data warehouse in a regular database, but you structure the data a little differently, and then you optimize for read versus for write.
Alexis
So, that’s a warehouse. I follow you there.
Data Dave
Right. A data lake is a more modern term, to a certain extent, and it has less structure to it.
So, typically, a data lake is a dumping ground. If you’re dealing with unstructured data, we’re collecting it all together in a lake. But we don’t necessarily know the purpose. We don’t necessarily have as much structure on it.
The other thing about a warehouse is you will create the structure of a warehouse first, and then you will populate that warehouse. The questions that you can answer are defined by the structure of the warehouse. So, you have the questions that you answer. Things like, “I want to do my annual financial reporting.” Okay. That’s a standard set of questions, standard set of reports to be generated. I can build a data warehouse to support those questions. And then I can populate that warehouse with the data from my transactional systems.
And now, I can answer my financial reporting needs very quickly when I need to, by querying my data warehouse.
Alexis
Right.
Data Dave
“What were all my transactions? What were all my orders? What’s our total book of business at the end of January?” Whatever it might be, I can answer that question very quickly.
A data lake is less structured. “Where am I going to put all my data so that I can answer some questions that I don’t know yet?”
Alexis
Okay, before you go on, I want to hop back to something you said early when we were talking about data lakes. You said it’s a place you often put “unstructured data”.
Data Dave
Right.
Alexis
Now, I might be completely off about this, so I want you to correct me. I have always put, like, images and videos and like, audio files in unstructured data categories.
Data Dave
Absolutely.
Alexis
Is that what unstructured data is?
Data Dave
Well, yes and no. Yes, that is unstructured data. That is absolutely unstructured data. But data without defined structure is also unstructured data. A bunch of spreadsheets that have no defined structure. Yes, they’ve got structure in and of themselves, but it’s not necessarily highly structured data.
Alexis
As in, it doesn’t have a larger governance protocol sitting over it, telling it what to do.
Data Dave
Exactly. Exactly. This isn’t a sales ledger. This isn’t a one-of-these. Right? A data warehouse often depends on the structure of saying, “This is a sales ledger or a whatever.” Whereas unstructured data doesn’t have the overarching structure. So, we’re being a little bit loose with some of the terms, which is video and audio and all that is unstructured data. Absolutely. So are documents, Word documents. So are spreadsheets. All this stuff is unstructured or semi-structured data.
And then you’ve got structured data, which is your transactional systems and your data warehousing.
Alexis
Yes. Okay.
Data Dave
All of this can go in your data lake.
Alexis
Because it’s fluid.
Data Dave
Because your data lake has no structure, doesn’t need a structure to function. Some of the big definitions that I often use are things like, with a data warehouse and a database, structure comes first. With a data lake, structure comes second, and it’s late-binding structure. So, we take the data, and then we layer the structure on top of it.
Alexis
Which sounds like a hot mess to me.
Data Dave
It is a hot mess.
Alexis
Okay, okay. Just wanted to put that out there.
Data Dave
It’s really a way of wrangling the hot mess.
Often, you say, “I’m going to put all my data here in my data lake, and then I’m going to read it. Well, how am I going to read it? I can only read it if I put structure on it.”
Alexis
Right.
Data Dave
So, this spreadsheet that is semi-structured contains a chart of accounts. Chart of accounts is very structured. So, somebody has manipulated this spreadsheet, used the chart of accounts, created a spreadsheet. Now, I’ve got a semi-structured spreadsheet that says something.
Alexis
Because part of it, that’s the chart of accounts is very structured, but the rest of it really isn’t.
Data Dave
The rest of it really isn’t. It’s kind of what somebody did with it, and they might have massaged and changed the chart of accounts or whatever they did. They do what they need to do.
When I put that into my lake, that’s fine. When I read that from an external point of view, I probably want to overlay the chart of accounts on top of it. So, I want to put that through some processing to extract the chart of accounts piece and use the true defined chart of accounts data. That’s what we call late binding.
I’m taking the spreadsheet, and I’m reading it through the lens of the chart of accounts. Yeah. That gives me the ability to read that spreadsheet as structured data through that chart of accounts. So, I’m late-binding the structure to the data. If I was putting that in a warehouse, I’d have to build it rigidly to the chart of accounts so that I could load it into the chart of accounts warehouse. And I can’t load it unless it conforms precisely to the chart of accounts.
Whereas in a data lake, what I’m saying is I’ve got the data, I’ve put the data in a “managed place.” And I’m reading it through the lens of the chart of accounts.
Alexis
No one saw you do air quotes when you said “managed”, but I want to tell everyone out there, Dave was like in an air quotes “managed place”.
Data Dave
Well, these data lakes aren’t always as managed as they should be.
Alexis
Right? As we said earlier – hot mess.
Data Dave
Yeah, exactly.
But the point is that it’s a centralized location. Often, it is data that is of interest to more than one person, and it is data that is being used probably for a purpose now, as it moves through the data lake.
I actually define “data lake” bigger. So, I will put areas of warehouses into my data lakes. So, I’ll have some very, very structured data and some semi-structured data in my data lake and use them in conjunction with each other, and that becomes part of the broader data lake environment.
Does that make sense?
Alexis
Did that make sense to you?
Find out if it makes sense to me in our next episode of Talk Tech with Data Dave. We turned this episode into a two-part episode because Dave and I had such a wonderful conversation about data lakes versus data warehouses, and I wanted to make sure we had enough time to dive deep into this concept.
So, join us in our next episode of Talk Tech with Data Dave, so we can finish our conversation about the difference between data lakes and data warehouses. And to maybe even learn about what Dave calls a “data pond”. Till next time. Thanks, everyone.
