The situation
Echo is an online pharmacy founded by patients for patients. With a strong customer centric approach, Echo is on a mission to help people better manage their health. Through Echo’s website or app, patients can order their medications in just a few clicks. Medications are then delivered to their door for free.
The company is 4 years old and has around 80 employees. They are also a fully operating pharmacy with a warehouse in West London.
Echo is a complex business and effective use of data is critical to pilot business operations and support strategic decisions The data team sits at the heart of the organisation ensuring good quality data and pertinent analysis are available for all, leading to better, faster decisions. Data is used for business operations, making sure that correct prescriptions are delivered on time and that sufficient stock is available in the warehouse. The product/engineering side of the business leverages data to identify gaps in the customer experience, prioritise projects and monitor releases.
Given the diversity and complexity of their data Echo needed a product to solve the following problems:
- Building a single source of truth for data across the organisation
- Ensuring data quality
- Building a data stack that would scale with the growth of the data team
- Helping the engineering and data teams to collaborate more closely
Building a single source of truth for data across the organisation
“Being able to have data that we can rely on has had a big impact on our productivity. It’s a huge time saver.”
All Echo's data is event-based. Every time an action occurs, (e.g. a change of status of an order) an event is generated. This means Echo have lots of data to work with, which is great, but it also means there is more room for error.
Often different analysts would create two different tables to answer the same question. This lead to different answers, and incorrect information being pushed into the wider business and product teams. Echo needed a single source of truth that they could use to answer questions
With Dataform, Echo was able to build a single source of truth for data across the entire organization using dependencies. For example, they now have a single user stats table that aggregates details on users activity. This can then be joined to any query that requires such details.There is no risk of getting two different metrics in answer to the same question.
Ensuring data quality
"We are now confident that data that goes out is solid and accurate, which wasn’t entirely the case before.”
Due to the complex nature of Echo’s datasets there were sometimes instances of errors getting pushed to downstream pipelines. Often it would be the business user who would spot these before the data team itself. Echo needed a tool that would help them catch data errors before they were pushed.
With Dataform, the team uses data assertions to catch errors in their pipeline before they are pushed downstream. If a build contains an error then it would fail and the team can fix it before it goes out to users! This means the team spends less time fighting data quality issues, and more time analysing data and helping the wider business focus on where to spend their time.
Building a stack that scales
“Migrating to Dataform and reorganising our data was a needed investment in order to keep growing the team and keep delivering value to the business.”
Before using Dataform, Echo used a mixture of BigQuery views and Looker PDTs to model their data, however they were finding this didn’t scale. The amount of data they were creating was growing very quickly they had over 140 tables in Looker (based on derived tables). This made it very hard to maintain a single source of truth. Echo needed a solution that could handle their growing datasets and a tool where the growing team could collaborate.
Using Dataform, Echo now has clear and managed table dependencies that allow for code reusability. This led to a 3x reduction in number of Looker views and no more PDTs making it far easier to maintain. This has given time back to the data team to work with other areas of the business on high value projects. Additionally, they have received great feedback from the wider business on how much easier data and metrics are to find.
Helping the engineering and data teams to collaborate more closely
“I think it’s a big step towards the famous data driven approach which is quite major.”
Echo found that the data and tech team would often work independently of each other. The engineering team would work on getting data into BigQuery and the data team would then query it from there. They wanted to build bridges between engineering and data by creating a more collaborative working environment.
Echo uses Dataform to adopt software engineering best practices e.g. code reviews and unit tests. They now find it easier to get advice from the engineering team as everything is linked to Github. They can commit changes, create pull requests and request code reviews from members of the engineering team.
Conclusion
Dataform allows Echo to have one place where the data team can collaborate on all things data! They now have reliable data which allows them to scale their data stack whilst building bridges with the engineering team.
"The support the Dataform team gave as well as all the help and best practices they shared was super valuable and helped us get to where we are now."
Echo weren't necessarily looking for a solution but after realising that their existing transformation layer wouldn't scale they decided to try out Dataform. The Dataform team helped Echo to think about how to re-organise their data and create a single source of truth that could be used to power analytics across their entire organisation.
A future with Dataform…
“We want to leverage more of the javascript capabilities, we think that’s a really cool feature!”
Dataform has allowed Echo to build a stable set of data tables that can be easily queried. They now want to take their data and use it to do predictive analysis - to improve customer experience, manage warehouse stock inventory and predict future orders.
Echo have also been using Dataform’s javascript capabilities to build a set of sequential user funnels. Previously, anyone could use Looker to build a funnel using events, however these were not sequential and could lead to misinterpretations. Dataform allowed Echo to build a set of sequentials funnels that are pre-defined and accurate for everyone to view in Looker.
Echo want to continue to leverage more of the javascript capabilities within Dataform to expand on the analytical capabilities of SQL.
