Chariot Solutions – Medium

Chariot Solutions

Published in
Thoughts on Data Engineering

Aggregating Files in Your Data Lake: Part 3

Several years ago, I wrote the post Friends Don’t Let Friends Use JSON (in their data lakes). The crux of that post was that JSON doesn’t…

Mar 27, 2024

Aggregating Files in Your Data Lake: Part 3

Mar 27, 2024

Published in
Thoughts on Data Engineering

Aggregating Files in your Data Lake: Part 2

In my last post, I developed a data pipeline to aggregate CloudTrail log files. When I ran this pipeline against Chariot’s CloudTrail…

Feb 29, 2024

Aggregating Files in your Data Lake: Part 2

Feb 29, 2024

Get an email whenever Chariot Solutions publishes.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Medium sent you an email at to complete your subscription.

Published in
Thoughts on Data Engineering

Aggregating Files in your Data Lake: Part 1

As I’ve written in the past, large numbers of small files make for an inefficient data lake. But sometimes, you can’t avoid small files…

Feb 15, 2024

Aggregating Files in your Data Lake: Part 1

Feb 15, 2024

Published in
Thoughts on Data Engineering

Data Engineering: More SRE Than SQL

Following my post about the Chariot Data Engineering interview, I received some comments along the lines of “wait, you don’t test their SQL…

Jan 9, 2024

Data Engineering: More SRE Than SQL

Jan 9, 2024

Published in
Thoughts on Data Engineering

Developing A Coding Test for Data Engineering

Hiring good candidates is difficult. After nearly 40 years in this business, and interviewing hundreds of candidates, I’m not going to…

Nov 21, 2023

Developing A Coding Test for Data Engineering

Nov 21, 2023

Published in
Thoughts on Data Engineering

Small Data: A Pipeline for Low-Latency Decision Support

In my last post, I said that I didn’t think Postgres was a good choice for a decision support database, versus a task-specific DBMS such as…

Aug 14, 2023

Small Data: A Pipeline for Low-Latency Decision Support

Aug 14, 2023

Published in
Thoughts on Data Engineering

Why Not Just Use Postgres?

My last few posts have focused on Redshift and Athena, two specialized tools for managing and querying Big Data. But there’s a meme that’s…

Jul 31, 2023

Why Not Just Use Postgres?

Jul 31, 2023

Published in
Thoughts on Data Engineering

A Deep Dive on Redshift Execution Plans

Execution plans are one of the primary tools to optimize your database queries, but they can be daunting to read and understand. In this…

Jul 17, 2023

A Deep Dive on Redshift Execution Plans

Jul 17, 2023

Published in
Thoughts on Data Engineering

Unbalanced Data in Redshift

I first experienced unbalanced data in a data warehouse thirty years ago. I was working for a mutual fund company, and something like a…

Jul 3, 2023

Unbalanced Data in Redshift

Jul 3, 2023

Published in
Thoughts on Data Engineering

Rightsizing Data for Athena

Amazon Athena is a service that lets you run SQL queries against structured data files stored in S3. It takes a “divide and conquer”…

Jun 19, 2023

Rightsizing Data for Athena

Jun 19, 2023

Chariot Solutions

Chariot Solutions

We solve your most complex business problems with our strategic, high-touch approach and extensive tech experience. Learn more at chariotsolutions.com

Following

Help
Status
About
Careers
Press
Blog
Privacy
Rules
Terms
Text to speech