Most existing analytics tools seem to be trying to build AI into their product, and the UI is mostly the "data chatbot". I'm personally more bullish of the AI tools that are being built from the ground up.
great insights! key take away is that generating hypotheses and applying judgment remains hard to automate - focus on how humans can enhance AI generated outputs to stay relevant
Wow this is an awesome mention of what we're building at Galaxy - https://www.getgalaxy.io. Really appreciate the shoutout and pumped to talk to your team :)
True data democratization *needs* to start with developers and we will be able to bridge the gap between question and answer better than anyone else that way!
I'm working on a project focused on answering natural language business questions, essentially converting text to SQL or Pandas queries.
I appreciate your balanced analysis of both bullish and bearish perspectives. I lean bearish, not due to a lack of technology, but because users often ask vague questions and lack clarity in their analysis objectives.
this to me is a big part of why you want a general purpose product serving all analytical tasks at a company. You have analysts, PMs, etc. who know how to ask very specific questions and tune the semantic model, and then everyone else gets to draft off of that
This is an amazing write-up. Left me wishing I had written it.
We're building open source AI data workers, our first being the analyst and are seeing the same things you're seeing.
For data teams, the semantic modeling and cataloging portion is the most difficult.
We're seeing that most data teams have a hard time anticipating the data that users actually care about. So, once they implement something like this, they are sprinting to build and clarify metrics, filters, etc. A good example, we have a customer that was requesting data around the location of their signups. Their top 5 results that came back were data center locations bc a lot of their users sign up with VPNs. Our agent investigated the data and found over 200k of signups from Ashburn, Virginia. Theres a huge data center there and the towns population is 45k. The agent actually filtered that out bc of the discrepancy, BUT it didn't filter out a data point in Chicago and Atlanta that also fit the same criteria.
That, frankly, is something that has to be documented or modeled so that the agent doesn't make the same mistake. Making that documentation process *fast* and *easy* is the biggest problem preventing teams from adopting AI analysts today. We're making a bet on automated documentation agents for this, but its still in the early stages.
For data users, the hypothesis portion is the most difficult to solve.
With access to even 10-20 datasets, there are theoretically an infinite number of hypothesis that can be made for different reasons. Usually, the *best data users or analysts* have good intuition about the hypothesis and context that help answer the right questions. When you can actually give a good descriptive prompt of the business context, your goals, the questions you're trying to ask, etc. I've seen our implementation do a great job of the rest of the analyst workflow when the hypothesis portion is well hashed out.
Less data-savvy folks tend to really struggle with the hypothesis and context, leading to pretty subpar results. One of our customers right now actually has users submit requests in slack, then they they reply with an @Buster and add in the additional context and hypothesis. That workflow alone helped them cover what they would've previously needed a FTE for.
Great context and very much agree on the semantic modeling part being difficult. I can't wait for these tools to make that part easier and allow people to focus more on the other phases.
Fun read, thanks for sharing! I'd really be shocked if we don't see the analysis agents happen sooner though especially after witnessing and experiencing Claude Code. It's crazy how quickly agentic agents are now spinning up PRs for many code repos and how founders are building agent armies so to speak for different use cases in the coding stack.
I think a similar workflow will happen in analysis where you have a human in the loop to help setup the system and leverage creativity but then the data agents will be able to do 95% of the grunt work.
I can also see a paradigm shift in which analytics and insights will be a "pull" not a "push" AKA as long as the shared AI knows what features are getting shipped or what is changing in the product or what KPIs matter to a startup -- they'll be surfacing reports, release analytics, etc all at the right time instead of us having to remember to do a release review or A/B test review -- this is why I've been thinking about the best way to do a shared/company changelog because I think it's crucial for this use case but also for any AI agentic workflow.
I do think that for more basic analysis that require less judgement (the two examples I give are deciding wether or not to ship an experiment and a weekly performance report) we can get to agentic faster. However if I think about the composition of the work that the team at Faire does, most it requires much more judgement and will be hard to get close to end-to-end for a while.
Have not read it yet but looks interesting and will check it out! What is your thesis? More or less transformational; faster/slower timeline than what I propose here?
give it a gander...it affirms elements of your theme/thesis but from a different angle....i draw from the karpathy's work as context & inspiration....again, check it out
This post hits the bull's eye on the problem and the opportunity. Thank you, Dan, for sharing your perspective. We at https://trust3.ai/platform/ have been tinkering with this problem for some time. Our Trust3 IQ product builds the semantic layer using AI, and we have found it to be far more accurate and efficient with the generated SQL and analysis of that data over existing incumbents that you have quoted here. If you’re up for it, we'd be happy to give you a demo and see if that resolves your data challenges at Faire.
Great post. Where does Amplitude fit in with their new agent offering in beta?
Most existing analytics tools seem to be trying to build AI into their product, and the UI is mostly the "data chatbot". I'm personally more bullish of the AI tools that are being built from the ground up.
great insights! key take away is that generating hypotheses and applying judgment remains hard to automate - focus on how humans can enhance AI generated outputs to stay relevant
🎯
Wow this is an awesome mention of what we're building at Galaxy - https://www.getgalaxy.io. Really appreciate the shoutout and pumped to talk to your team :)
True data democratization *needs* to start with developers and we will be able to bridge the gap between question and answer better than anyone else that way!
Excited to learn more about what you are building
I'm working on a project focused on answering natural language business questions, essentially converting text to SQL or Pandas queries.
I appreciate your balanced analysis of both bullish and bearish perspectives. I lean bearish, not due to a lack of technology, but because users often ask vague questions and lack clarity in their analysis objectives.
this to me is a big part of why you want a general purpose product serving all analytical tasks at a company. You have analysts, PMs, etc. who know how to ask very specific questions and tune the semantic model, and then everyone else gets to draft off of that
This is an amazing write-up. Left me wishing I had written it.
We're building open source AI data workers, our first being the analyst and are seeing the same things you're seeing.
For data teams, the semantic modeling and cataloging portion is the most difficult.
We're seeing that most data teams have a hard time anticipating the data that users actually care about. So, once they implement something like this, they are sprinting to build and clarify metrics, filters, etc. A good example, we have a customer that was requesting data around the location of their signups. Their top 5 results that came back were data center locations bc a lot of their users sign up with VPNs. Our agent investigated the data and found over 200k of signups from Ashburn, Virginia. Theres a huge data center there and the towns population is 45k. The agent actually filtered that out bc of the discrepancy, BUT it didn't filter out a data point in Chicago and Atlanta that also fit the same criteria.
That, frankly, is something that has to be documented or modeled so that the agent doesn't make the same mistake. Making that documentation process *fast* and *easy* is the biggest problem preventing teams from adopting AI analysts today. We're making a bet on automated documentation agents for this, but its still in the early stages.
For data users, the hypothesis portion is the most difficult to solve.
With access to even 10-20 datasets, there are theoretically an infinite number of hypothesis that can be made for different reasons. Usually, the *best data users or analysts* have good intuition about the hypothesis and context that help answer the right questions. When you can actually give a good descriptive prompt of the business context, your goals, the questions you're trying to ask, etc. I've seen our implementation do a great job of the rest of the analyst workflow when the hypothesis portion is well hashed out.
Less data-savvy folks tend to really struggle with the hypothesis and context, leading to pretty subpar results. One of our customers right now actually has users submit requests in slack, then they they reply with an @Buster and add in the additional context and hypothesis. That workflow alone helped them cover what they would've previously needed a FTE for.
Great essay. Thanks for sharing.
Great context and very much agree on the semantic modeling part being difficult. I can't wait for these tools to make that part easier and allow people to focus more on the other phases.
Fun read, thanks for sharing! I'd really be shocked if we don't see the analysis agents happen sooner though especially after witnessing and experiencing Claude Code. It's crazy how quickly agentic agents are now spinning up PRs for many code repos and how founders are building agent armies so to speak for different use cases in the coding stack.
I think a similar workflow will happen in analysis where you have a human in the loop to help setup the system and leverage creativity but then the data agents will be able to do 95% of the grunt work.
I can also see a paradigm shift in which analytics and insights will be a "pull" not a "push" AKA as long as the shared AI knows what features are getting shipped or what is changing in the product or what KPIs matter to a startup -- they'll be surfacing reports, release analytics, etc all at the right time instead of us having to remember to do a release review or A/B test review -- this is why I've been thinking about the best way to do a shared/company changelog because I think it's crucial for this use case but also for any AI agentic workflow.
Anyways, that's my rant -- thanks again.
I do think that for more basic analysis that require less judgement (the two examples I give are deciding wether or not to ship an experiment and a weekly performance report) we can get to agentic faster. However if I think about the composition of the work that the team at Faire does, most it requires much more judgement and will be hard to get close to end-to-end for a while.
Interesting - Hypothesizing will become even more important. Time to put my PM hat on
nice post.... those interested in 'vibe analytics' MIT Sloan Management Review's origins might want to check out https://sloanreview.mit.edu/article/vibe-analytics-vibe-codings-new-cousin-unlocks-insights/. well worth a look (says its author immodestly...)
Have not read it yet but looks interesting and will check it out! What is your thesis? More or less transformational; faster/slower timeline than what I propose here?
give it a gander...it affirms elements of your theme/thesis but from a different angle....i draw from the karpathy's work as context & inspiration....again, check it out
This post hits the bull's eye on the problem and the opportunity. Thank you, Dan, for sharing your perspective. We at https://trust3.ai/platform/ have been tinkering with this problem for some time. Our Trust3 IQ product builds the semantic layer using AI, and we have found it to be far more accurate and efficient with the generated SQL and analysis of that data over existing incumbents that you have quoted here. If you’re up for it, we'd be happy to give you a demo and see if that resolves your data challenges at Faire.